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Executive Summary 



New Jersey After 3 (NJ After 3) is a private, nonprofit organization whose mission is to 
expand and improve afterschool opportunities for New Jersey’s youth. Using both public and 
private resources, it supports a network of youth-service providers that deliver afterschool 
services based on NJ After 3’s program model. The NJ After 3 model aims to increase the 
number of afterschool programs in New Jersey that provide a safe environment for children 
during afterschool hours, offer enriching academic activities and homework assistance, and 
expose children to nurturing individuals and meaningful experiences that promote intellectual, 
physical, social, and artistic development. Among the tools that NJ After 3 uses to achieve its 
mission, in addition to grantee selection and financial support, are professional development 
opportunities extended to site coordinators and program staff, special curricular modules to 
address high-priority youth needs (e.g., physical fitness, math problem-solving), technical 
assistance for program improvement, and specialized software for youth-level tracking of 
enrollment and attendance. 

Under an agreement with NJ After 3, Policy Studies Associates, Inc., has conducted a 
longitudinal evaluation of the initiative’s operations and outcomes. The evaluation has assessed 
the extent to which NJ After 3 has expanded the availability of afterschool services, facilitated 
the delivery of high-quality program content, produced positive outcomes for youth, and created 
sustainable systems for funding and program quality. Evaluators collected data over three 
program years, beginning in 2005-06, which was the second year of NJ After 3 service delivery. 

As described in this report, NJ After 3 achieved significant success on the four measures 
at the heart of this study. With its partners, it expanded the availability of afterschool services, it 
facilitated the achievement of quality in both service delivery and program content, it produced 
positive outcomes for participating youth, and it built sustainable systems of program operations 
and financial support. 



Program Availability 

NJ After 3 increased overall enrollment in its afterschool program from approximately 
4,000 participants in grades K-8 in the first year of the initiative to almost 15,000 participants in 
2007-08, the evaluation’s final year of data collection. The background characteristics of 
participants changed little during the period of the evaluation, with the program consistently 
attracting more participants from the early grades. Forty-five percent of participants in 2007-08 
were in grades K-2, compared with 35 percent and 19 percent of participants in grades 3-5 and 
grades 6-8, respectively. Feedback from program staff, parents, and participants suggested that 
this difference is likely a result of two important factors. First, older participants often had 
opportunities to participate in team sports that are offered by other organizations but are not 
typically available at NJ After 3 programs. In addition, working parents needed the program to 
supervise their younger children after school, but parents tended to permit the older children to 
remain at home without adult supervision during the afternoon. Also, new programs initiated in 
Newark during the evaluation period concentrated services in the early grades. The percentage 
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of participants who were in grades K-2 increased over the three-year evaluation period. 
Compared with 2005-06, the evaluation’s first year, the 2007-08 figures represent increases 
among participants in grades K-5, and a small decrease among participants in grades 6-8. 

Not only did more youth attend NJ After 3 programs over time, but enrolled youth 
attended more often. A total of 107 programs reported participants’ attendance data for 2007-08 
and, overall, youth attended 115 days on average in that year, compared with 94 days in 2005-06. 
The average attendance rate increased at all grade levels by an average of 8 percentage points, 
from 73 percent to 81 percent over the evaluation’s three years. The largest increase was among 
youth in grades 6-8. The attendance rate for youth in those grades increased by 9 percentage 
points (from 62 percent to 7 1 percent), compared with a 6 percentage-point increase among 
youth in lower grades. Thus, while the number of older youth enrolled in NJ After 3 decreased 
slightly over the three years, those who were enrolled in the evaluation’s last year attended more 
frequently. Their increased attendance suggests that youth were interested in and engaged by the 
activities provided and that they had positive experiences with staff and peers in the programs. 

The attendance rate of 1 15 day per year (81 percent) among NJ After 3 participants 
compares favorably with other afterschool programs. For example, the evaluation of the 
CORAL afterschool programs in California revealed that, on average, youth in those programs 
attended approximately 110 days (with an attendance rate of 73 percent) during the program 
year. The national evaluation of 2 1 st Century Community Learning Centers found that youth in 
the evaluation sample attended that federal program 58 days on average per year, with 
attendance rates of 37 percent in the elementary grades and 18 percent in the middle grades. 

Although more youth enrolled in NJ After 3 programs and attended more often over time, 
their retention rate (i.e., the percentage who attended a NJ After 3 program in one year, were 
eligible to return to the same program the next year, and actually did so) declined slightly over 
time. In 2006-07, the retention rate was 5 1 percent, and in 2007-08 it was 46 percent. Although 
lower than the previous year’s rate, the 2007-08 retention rate exceeded the 36 percent retention 
level of a similar large-scale afterschool initiative in New York City. It is unclear whether the 
two-year decline in the NJ After 3 retention rate is the result of increased mobility among the 
families of participants, limited enrollment capacity within individual programs, or some other 
reason. 



Encouragement of Youth Outcomes 

Teachers identified important academic outcomes among students who participated in NJ 
After 3 in academic engagement and interpersonal skills, study habits, reading and language arts, 
and technology-related skills during each year of the evaluation. This cluster of skills is 
commonly believed to constitute essential cognitive abilities needed to navigate the career and 
citizenship demands of the 21 st Century. Teachers reported in 2007-08 that over three-quarters 
of program participants were at grade level or above in their ability to draw conclusions from 
written materials, spell, identify the main idea in a passage of reading, and communicate orally. 
Teachers also reported that most participants “almost always” or “often” demonstrated the key 
study skills listed in the evaluation’s survey. Similarly, they reported that most participants’ 




technology skills were good or excellent in comparison to other students at their school, 
particularly in using word processing programs, spreadsheet programs, and the Internet. Older 
students in particular reported that they had benefited from access to technology in the 
afterschool program. 

In order to assess change over time in the academic skills of program participants whose 
skills had been rated by their teachers over a two-year period, evaluators identified 86 NJ After 3 
participants who had been rated by a teacher in any two consecutive years of the three-year 
evaluation period. Ratings of these students’ reading and language arts skills rose from a scale 
score of 2.79 (on a live-point scale) in the first year to a scale score of 3. 14 in the second year, 
which represents a significant learning gain (effect size=.24). 

Most participants believe that participation in NJ After 3 has helped them academically. 
In 2007-08, they reported that they “agreed” or “agreed a lot” that the program helped them to 
finish their homework more often, get better grades in school, read and understand better, solve 
math problems better, write better, and use computers to do schoolwork better. 

A large majority of participants reported that they generally enjoyed their experiences in 
the afterschool program. They reported, for example, that in the program they felt safe and that 
they felt that the program was a good place to spend their afternoons. In addition, they believed 
that staff think they can do things well, treat them with respect, try to be fair, think they can leam 
new things, and care about them. Participants also indicated that they enjoyed positive 
relationships with peers in the program. A majority reported that they had many friends in the 
program, they had a good time playing with other youth in the program, and, in the program, 
they get to work with others as part of a team. 

Parents interviewed in the evaluation were satisfied with their children’s NJ After 3 
experiences and outcomes. Many indicated that their children were introduced to new activities 
and classes that they could not otherwise have experienced, due to the family’s financial 
constraints, and that their children were given increased opportunities to socialize and play with 
their peers through the program. Several also said that their children began to demonstrate new 
or improved skills in reading and math as well as improvements in social skills and attitudes 
toward school after they enrolled in NJ After 3. 



Program Content and Quality 

Evaluators examined the content and quality of program services from many vantage 
points and produced descriptive findings across the initiative as a whole and at the level of 
individual programs. Looking most broadly, analyses revealed that NJ After 3 programs 
provided a wide range of activities to address the academic, social, cultural, and recreational 
interests and needs of participants. During 2007-08, according to data recorded in the initiative’s 
central data collection and reporting system, the average NJ After 3 participant experienced the 
following hours of programming, as reported by programs and using the activity categories 
established by NJ After 3: 
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■ 54 hours of health and sports activities (e.g., nutrition, basketball, cooperative 
games) 

■ 41 hours of visual arts and perfonning arts activities (e.g., drawing, dance, drama) 

■ 38 hours of community service/character education activities (e.g., violence 
prevention, instruction in resisting peer pressure, food drives) 

■ 33 hours of literacy learning and practice (e.g., reading groups, spelling bees, 
journal reflections) 

■ 32 hours of study skills and college exploration (e.g., test preparation, tutoring, 
trips to colleges) 

■ 25 hours of math activities, including math applications (e.g., financial literacy, 
KidzMath) 

■ 19 hours of technology and science-related activities (e.g., computer and video 
projects, earth science) 

■ 19 hours of cultural activities (e.g., politics, community exploration) 

■ 18 hours of “New Jersey” (e.g., New Jersey history, culture, geography) 

Evaluators conducted detailed observations of 78 activities at 10 programs in 2007-08. 

The observation ratings yielded scale scores that were then aligned with key features of effective 
afterschool programs, as described by Durlak and Weissberg in their meta-analysis of 73 
afterschool programs (2007). The scale scores ranged from 1 (indicator is not evident) to 7 
(indicator is highly evident and consistent) The analysis revealed that, in general, activities 
observed in 2007-08 were rated 3.64 for “sequenced” practices that built skills and content 
knowledge to achieve goals; 1.83 for “active” practices that offered opportunities to actively 
participate in learning; 4.40 for “focused” practices that developed positive relationships among 
youth and with staff; and 5.21 for “explicit” practices that targeted specific learning goals and or 
developmental goals. 

All activities scored high on the “focused” and “explicit” scales, indicating that activities 
were often structured to develop personal and social skills in youth and featured specific learning 
goals. They tended, however, to score relatively low on the “active” scale, suggesting that 
programs provided relatively few opportunities for hands-on practice and the application of skills. 

Analyses revealed statistically significant relationships in three areas that may warrant 
further consideration: 

■ The extent to which participants reported that they benefited academically by 
participating in the program was positively associated with the availability of 
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community service projects (r=0.86, effect size=0.73) and art activities (r=0.57, 
effect size=0.32). 

■ The extent to which participants reported that the program provided them with 
exposure to new opportunities was positively associated with the availability of 
academically-oriented activities that supported literacy skills development, 
cognitive development, and academic achievement, as reported by site 
coordinators ( 7=0. 50, effect size=0.25). 

Evaluators created a study skills scale that included teachers’ assessments of how well or 
how often students prepared for tests, took notes in class, completed homework, corrected their 
own work, and paid attention in class. Participants who attended the program regularly (i.e., 
those who attended at least 80 days a year and attended at least 80 percent of the days they were 
enrolled) were rated on the scale at 3.8, out of a possible 5.0. This score was significantly higher 
than the mean scale sore of 3.5 for participants who attended less frequently (i.e., those who 
attended fewer than 60 days or less than 60 percent of the time in which they were enrolled) 
(effect size=0.15). This relationship between NJ After 3 participation level and teacher-rated 
study skills suggests the possibility that learning experiences in the NJ After 3 program 
positively influenced students’ study skills. 

A majority of parents and youth reported that the afterschool programs provided by NJ 
After 3 met their needs. Services were in demand within participating schools, with some NJ 
After 3 sites maintaining waiting lists. 

All or nearly all of the observed activities met basic quality thresholds, including the 
maintenance of adequate staff/student ratios and the provision of adequate space and materials. 

A subset of programs, however, offered activities and classes that didn’t fully meet other 
important goals and objectives, such as actively engaging youth in learning or providing 
intellectually challenging content. Site coordinators described several factors that limited 
program quality. These included poorly qualified staff in some sites. In addition, some site 
coordinators reported limited success in gaining access to sufficient resources from host schools, 
the community, and other potential donors to provide the full range of experiences, classes, and 
activities that they believed were important for well-rounded development. 



Program Sustainability 

Staff at several programs indicated that they were challenged to sustain current levels of 
service. Reduced funding from NJ After 3 (based on program parameters set when NJ After 3 
was established) had resulted in increased program fees and or a reduction in programming at 
some sites. Although programs knew in advance that funding from NJ After 3 would be 
gradually reduced over time, some programs reported that they had not developed strategies for 
securing new revenue. In the evaluation’s surveys, 41 percent of site coordinators identified 
insufficient funds as a major challenge, and 40 percent identified it as a minor challenge. 
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NJ After 3 currently provides many professional-development, curricular, and technical 
assistance supports to programs, including information and training regarding available resources 
and funding opportunities. Feedback from programs suggested that even more assistance is 
needed to help sites develop new means of generating revenue. 
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1. Overview of the Evaluation and This Report 



Under an agreement with New Jersey After 3 (NJ After 3), Policy Studies Associates, 
Inc., conducted a comprehensive evaluation of this statewide afterschool initiative. This report, 
based on data collected in program years 2005-06, 2006-07, and 2007-08, assesses the extent to 
which the initiative is making progress toward achievement of the following goals: 

■ Expanding the availability of afterschool services in New Jersey 

■ Enhancing the quality of afterschool services 

■ Creating sustainable local programs that are financially stable and of high quality 

■ Promoting the health and the emotional, social, and intellectual development of 
New Jersey’s children 

Throughout the report, we present certain data collected in the evaluation’s third year 
(program year 2007-08) and first year (program year 2005-06) in tables and graphs in order to 
illustrate important program changes, developments, and outcomes. In some instances, the 
changes are statistically significant. All statistically significant changes are highlighted within 
the exhibits, and they are also noted in the text of the report. In addition to highlighting changes 
between these years, the report describes factors and conditions that facilitated or impeded 
program effectiveness, and it recommends strategies that can improve program implementation 
and outcomes in the future. 

This report is organized in six main sections. The first section describes the evaluation 
design and processes. Section 2 describes program participants, the frequency with which they 
attended NJ After 3, and their year-to-year retention rates. This section also examines challenges 
associated with recruiting and retaining youth at targeted grade levels. Section 3 describes the 
non-academic (i.e., social, emotional, behavioral, and attitudinal) and academic outcomes of 
program participants. Section 4 describes the activities that are offered by NJ After 3. Section 5 
highlights important program accomplishments and examines the challenges that impede 
program implementation and program quality. Section 6 outlines recommendations for 
increasing sustainability and securing the resources necessary to increase program reach and 
ensure program quality. 

Within each section of the report, where appropriate, we also compare and contrast 
certain evaluation findings with comparable findings from evaluations of similar afterschool 
programs in other metropolitan areas. Specifically, we highlight similarities and differences in 
program features and outcomes as assessed in this evaluation of NJ After 3 and in evaluations of: 
the New York City Department of Youth and Community Development’s Out-of-School Time 
Programs for Youth initiative (DYCD-OST) (Russell, Mielke, & Reisner, 2008); programs 
sponsored by The Afterschool Corporation (TASC) in New York City (Reisner, White, Russell, 
& Birmingham, 2004); LA’s Best, a city-wide afterschool initiative in Los Angeles (Huang, 
Gibbons, Kim, Lee, & Baker, 2000; Huang, Kim, Marshall, & Perez, 2005); and the CORAL 



1 




Afterschool Program (Arbreton, Sheldon, Bradshaw, & Goldsmith, 2008), which operates in five 
California cities. We also provide comparisons to findings of the national evaluation of the 
federal 21 st Century Community Learning Centers Program (James-Burdumy, Dynarski, Moore, 
Deke, Mansfield, & Pistorino, 2005). Important differences among these studies could have 
contributed to differences in some of the reported results. For example, survey questions were 
not identical, even though they assessed similar domains and topics, and targeted participants 
varied in age and grade among the programs (e.g., NJ After 3 participants were in grades K-8, 
DYCD-OST participants included high school students, and CORAL participants were primarily 
in grades 1-5). Nevertheless, while not ideal, these comparisons are helpful in providing context 
and a point of reference for this evaluation’s findings. 



Overview of Evaluation Design 

In 2007-08, we collected data through the following activities, all of which had also been 
employed in prior years of the study: 

■ Data from NJ After 3’s management information system. We retrieved data 
on the characteristics and program attendance patterns of all NJ After 3 student 
participants from YouthServices.net, the vendor for the NJ After 3 management 
information system. 

■ Surveys of executive directors. Executive directors of local sponsoring 
organizations provided data on program goals, operations, structures, budgets, and 
challenges. 

■ Surveys of site coordinators. Site-level coordinators provided data on program 
goals and activities, program schedules, staff recruitment and qualifications, 
participant outreach and recruitment, participant needs and preferences, and 
efforts to make connections among participants’ schools, communities, and 
families. 

■ Surveys of student participants. We surveyed participants in grades 3-8 who 
attended programs whose first year of NJ After 3 support was school year 2004-05. 
These surveys yielded information on participants’ behaviors, attitudes, and skills. 

■ From a sample of programs that began operations in 2004-05 (known as the in- 
depth study sample), we collected the following types of data: 

■ Interview and observation data. Site visits to the in-depth study sites 
pennitted the collection of interview data from site coordinators, program 
staff, student participants, and parents, as well as information on 
programming. We also collected data through structured observations of 
program activities. 
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■ Survey data from school-day teachers. School-day teachers in nine in- 
depth study sites provided data on the behaviors, attitudes, and skills of NJ 
After 3 student participants. 1 We administered this survey in nine of the 
ten sites because only nine programs served participants in the grades 
targeted for the teacher survey in 2007-08. Teachers who could address 
participants’ general academic and reading/language arts skills were 
selected for survey response. In the evaluation’s first year, the survey was 
administered to teachers of participants in grades 3-6. In the second year, 
surveys were administered to teachers of participants in grades 4-7. We 
surveyed teachers of participants in grades 5-8 in the third year. 

Using these data sources, the evaluation addressed these primary research questions: 

1 . What are the characteristics of the programs supported by the NJ After 3 
initiative? 

2. What are the characteristics of the youth served by NJ After 3, and what are their 
patterns of attendance? 

3. What did NJ After 3 accomplish with respect to enhancing the quality of 
afterschool services? 

4. What did NJ After 3 accomplish with respect to expanding the availability of 
afterschool services? 

5. How successful was NJ After 3 in creating sustainable systems for funding and 
program quality? 

6. What were the initiative’s effects on participating children and families? 

Efforts to Obtain Informed Parental Consent 

In compliance with the Family Educational Rights and Privacy Act and PSA’s federally 
approved Institutional Review Board and to protect the privacy of all youth participants, we 
required that parental research consent be obtained prior to our surveying or interviewing any 
participant or our surveying a teacher about a participant. We prepared the parental consent form 
in English and Spanish, and NJ After 3 posted both versions on the NJ After 3 website for 
programs to download. In addition, NJ After 3 required all programs to include evaluation 
consent forms in their enrollment packets and to enter each participant’s parent/guardian 
evaluation-consent status into the appropriate field in the YouthServices.net tracking system. 



1 The teacher survey adapted and used certain items from the Academic Competence Evaluation Scales (ACES), 
which was developed by James DiPerna and Stephen Elliott for Harcourt Brace. 
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■ From programs that began operations in 2004-05, site coordinators obtained 
consent forms from parents or guardians of 82 percent of participants in grades 3- 
8 (1,546 students). Of these programs’ 1,953 participants in grades 3-8 with data 
in YouthServices.net, the parents or guardians of 1,383 participants gave consent, 
while parents or guardians of 215 participants denied consent. No consent data 
were recorded for 355 participants in grades 3-8. 

■ From the 10 in-depth study programs, the evaluation obtained consent fonns from 
93 percent of the participants in grades 3-8 (881 of the 950 students for whom 
consent was requested). Parents or guardians of 803 of the 881 participants who 
returned consent forms gave consent, while parents or guardians of 78 participants 
denied consent. No consent data were recorded for 69 of the 950 students for 
whom requests were made. 



Evaluation Data Used in This Report 

This report presents analyses of data collected from programs operating with NJ After 3 
support since 2004-05 (Round I programs), programs funded since 2005-06 (Round II 
programs), programs funded since 2006-07 (Round III programs), and programs funded for the 
first time in 2007-08 (Round IV programs). In 2007-08, the evaluation collected the following 
from all programs in Rounds I, II, III and IV : 

■ Survey data from 44 executive directors of provider organizations that sponsor NJ 
After 3 programs (98 percent of the 45 executive directors) 

In the evaluation’s first year, we collected survey data from 27 executive directors 
of provider organizations (69 percent of 39 executive directors). This survey was 
not administered in the second year. 

■ Survey data from 98 site coordinators (92 percent of the coordinators of 107 
programs), including 20 Round I site coordinators, 38 Round II site coordinators, 
23 Round III site coordinators, and 17 Round IV site coordinators 

In the evaluation’s first year, we collected survey data from 83 percent of site 
coordinators of 55 programs, and in the second year we collected survey data 
from 92 percent of site coordinators of 90 programs. 

■ Program attendance and demographic data on 14,919 program participants in 
grades K-8 from the YouthServices.net management infonnation system 

In the evaluation’s first year, we collected attendance and demographic data on 
1 1,108 participants and, in the second year, on 14,378 participants. 

From Round I programs only, we collected the following: 
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Survey data from 955 youth participants in grades 3-8 (72 percent of 1,326 
participants) 



In the evaluation’s first year, we collected survey data from 671 participants (73 
percent of 919 participants) and, in the second year, from 1,046 participants (68 
percent of 1,534 participants). 

We collected the following from the in-depth study sample of 10 Round I programs: 

■ Survey data from school-day teachers who assessed 204 NJ After 3 student 
participants in grades 5-8 with parental consent (48 percent of 425 participants) 

In the evaluation’s first year, we collected 303 surveys (57 percent of 527 
participants), and in the second year, the evaluation collected 375 surveys (61 
percent of 614 participants). 

■ Interview data from 10 site coordinators, 25 program staff, 36 youth, and 20 
parents 

In the evaluation’s first year, interview data were collected from 10 site 
coordinators, 57 program staff, 63 youth, and 23 parents. In the second year, 
interview data were collected from 10 site coordinators, 35 program staff, 46 
youth, and 2 1 parents. 

■ Observation data from 78 afterschool classes or activities during site visits to each 
of 10 programs 

In the evaluation’s first year, we conducted 179 observations, and in the second 
year, we conducted 159 observations. 

In general, statistical differences presented in this report were statistically significant at 
the p<.05 or the p<.01 level of significance. Analyses were tested based on either a chi-square 
test, a Pearson’s correlation (r), or an ANOVA test with Bonferroni post-hoc tests. For each 
measure reported, the evaluation also computed an effect size to describe the standardized 
magnitude of the difference. 2 For continuous variables, effect size was computed as measured 
by Cohen’s d (the difference in means divided by the pooled standard deviation); for 
dichotomous variables, the effect size was computed using Cramer’s V. There is no single 
accepted standard for interpreting an effect size, and the current custom is to consider the effect 
size in light of previous research. Previous afterschool studies generally reveal small effect sizes 
for student outcomes. In a review of four studies of afterschool programs, Kane (2004) 
concluded that the expected impact of an extra hour of instruction delivered in an afterschool 
setting over a school year equals an effect size of 0.05 in reading and math. In this report, we 



2 Statistical significance indicates how reliable an estimated effect is. Effect size indicates the relative magnitude of 
an estimated effect. Unlike a test of statistical significance, an effect size is not affected by the number of subjects 
in a sample. 
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interpret an effect size of 0.10 or higher as meaningful. Effect sizes below this threshold, even if 
statistically significant, are not deemed to represent meaningful effects. 

Examples of effect size reported in other recent research relative to this evaluation 
include the following: 

■ A study of the impact of the reduction in class size in elementary classrooms by 
eight students per class found an effect size of 0.23 on math scores after one year 
(Finn & Achilles, 1999). 

■ A study of the impact of instruction by Teach for America teachers on math 
achievement found an effect size of 0. 15 on math scores after a year of 
participation in a classroom led by a Teach for America teacher (Decker, Mayer, 
& Glazennan, 2004). 

■ An evaluation of the 2 1 st Century Community Learning Centers Program in 
Louisiana found that the impact of this afterschool program had an impact with an 
effect size of 0. 13 on a combined measure of reading, math, and language test 
scores (Jenner & Jenner, 2007). 
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2. Program Reach and Participation 



The NJ After 3 program serves a diverse youth population within its many host schools 
and communities. As illustrated in Exhibit 1, almost all site coordinators reported using open 
enrollment at their site, and, in comparison with 2005-06, smaller percentages of site 
coordinators reported targeting specific types of youth for enrollment in 2007-08. For example, 
significantly fewer coordinators (32 percent in 2007-08 and 47 percent in 2005-06) reported that 
they specifically target youth who scored “below proficient” on district or state assessments. 

Exhibit 1 

Types of Participants Targeted by NJ After 3 Programs, in Percents, 

2005-06 and 2007-08 




Exhibit reads: In 2007-08, 97 percent of site coordinators reported that their afterschool program had open 
enrollment for all interested youth, compared with 93 percent in 2005-06. The difference was not statistically 
significant. 

* Difference is significant at p< .05 
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Characteristics of Participants 



NJ After 3 has increased its overall enrollment significantly since 2005-06. As seen in 
Exhibit 2, the program enrolled almost 15,000 total students in grades K-8 during 2007-08, an 
increase from approximately 1 1,000 students in 2005-06 and from the approximately 4,000 
youth whom NJ After 3 reports that it served in 2004-05 (evaluators do not have independent 
corroboration of 2004-05 enrollment). Compared with other afterschool programs such as the 
DYCD-OST Programs for Youth, which serves more than 70,000 youth, New Jersey After 3 
reaches a smaller student population. However, NJ After 3 is demonstrating a pattern of growth 
similar to that of DYCD-OST. Enrollment in DYCD-OST programs increased 35 percent from 
2005-06 to 2006-07, while NJ After 3 programs served nearly 30 percent more students during 
the same period (Russell et ah, 2008; Walking-Eagle, Miller, Reisner, LaFleur, Mielke, Edwards, 
& Farber, 2008) 



Exhibit 2 

Demographic and Educational Characteristics of Participants, in Percents, 

2005-06 and 2007-08 



Characteristics 


2005-06 

(n=1 1 ,1 08) 


2007-08 

(n=14,91 9) 


Gender 






Male 


50 


50 


Female 


50 


50 


Race/Ethnicity 






African American 


49 


50 


Hispanic 


34 


37 


White 


15 


10 


American Indian or Native American 


1 


1 


Asian or Pacific Islander 


1 


1 


Free or Reduced Price Lunch 






Yes 


53 


57 


No 


17 


18 


Unspecified 


30 


25 


Limited English Proficiency 






Yes 


12 


12 


No 


60 


64 


Unspecified 


28 


24 


Special Education Status 






Yes 


2 


3 


No 


52 


52 


Unspecified 


56 


45 



Exhibit reads: Fifty percent of student participants were male in both years of the initiative. 
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While overall enrollment in NJ After 3 programs increased by nearly 4,000 youth from 
the evaluation’s first year to its third year, the increase in participants is primarily due to an 
expansion in the number of programs, rather than an increase in the number of participants being 
served in each program. Approximately one -third of the enrollment increase (1,283 youth out of 
3,818) reflects the expansion of existing programs in Newark and the start-up of new Newark 
sites. The statewide enrollment of the 65 2005-06 programs decreased by an average of 15 
students per program between the evaluation’s first and third years. The total enrollment of these 
65 programs decreased from over 1 1,000 to about 10,000 across this period, while enrollment in 
all NJ After 3 programs increased from over 1 1,000 to almost 15,000 during the same period. 



Participant Characteristics 

We detennined the demographic and educational characteristics of NJ After 3 participants in 
order to assess the likelihood that participants would be likely to benefit from afterschool services. 
We used standard reporting categories to describe participants’ background characteristics. The 
characteristics of NJ After 3 participants remained consistent between the first and third years of 
the evaluation, as illustrated in Exhibit 2. The initiative continued to serve equal numbers of boys 
and girls, who were predominately African American or Hispanic (87 percent of total participants 
in 2007-08). The proportion of participants who qualified for free or reduced-price lunches was 57 
percent. A minority of participants were English Language Learners (12 percent). Few 
participants received special education services (3 percent). The actual proportion of free or 
reduced-price lunch qualifiers, English Language Learners, and special education students may 
vary from reported figures, since these characteristics were not specified in the YouthServices.net 
database for a substantial number of participants. For example, eligibility for free or reduced-price 
lunch was not recorded for 25 percent of youth in 2007-08; other research suggests that a high 
proportion of these youth would be eligible for federally subsidized meals. 



Grade Distribution 

New Jersey After 3 programs attracted participants who were mainly in the early grades, 
as seen in Exhibit 3. Forty-five percent of participants in 2007-08 were in grades K-2. The 
proportion of participants in K-2 increased nine percentage points from the evaluation’s first year 
to its third year. In part, this shift reflects the increase in program slots in Newark serving youth 
in grades K-2; Newark’s K-2 enrollment grew by 855 youth from 2005-06 to 2007-08. The total 
number of participants in grades 6-8 declined by 149 youth statewide (from 3,025 to 2,876 
between the evaluation’s first and third year). 
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Exhibit 3 

Distribution of Enrolled Youth, by Grade Span, 2005-06 and 2007-08 



Grade 


Percent of 

NJ After 3 Total: 2005-06 

(n=1 1,108) 


Percent of 

NJ After 3 Total: 2007-08 

(n=1 4,91 9) 


Percentage-Point Change 


K-2 


36 


45 


+9 


3-5 


37 


35 


-2 


6-8 


27 


19 


-8 



Exhibit reads: Thirty-six percent of the youth in the NJ After 3 initiative were in grades K-2 in 2005-06, compared to 
45 percent in 2007-08. This change represented a nine percentage point increase. 



Attendance of Participants 

The NJ After 3 evaluation measured youth attendance for two reasons. First, regular 
attendance is a good reflection of parents’ and youth satisfaction with afterschool programming. 
Second, research shows that regular afterschool attendance is associated with significant 
educational and social benefits for participants (e.g., Huang et ah, 2000). The high levels of 
youth attendance in NJ After 3 programs put them in the forefront of afterschool programs 
nationally on this measure, as shown below. 



Days Attended 

A total of 107 programs reported participants’ attendance data for 2007-08. Participants 
on average attended more program days in 2007-08 than in 2005-06 across the three grade spans, 
as summarized in Exhibit 4. Overall, youth attended 1 15 days on average in 2007-08. In 
comparison, the evaluation of the CORAL afterschool programs in California (Arbreton et ah, 
2008) reported that, on average, youth in those programs attended approximately 110 days 
during the year. In the federal 21 st Century Community Learning Centers program, participants 
attended for 58 days on average over the course of the year (James-Burdumy et ah, 2005). 

Exhibit 4 

Distribution of Days Attended, by Grade, 2005-06 and 2007-08 



Grade Span 


Average Days Youth 
Attended in 2005-06 


Average Days Youth 
Attended in 2007-08 


Average Change 
in Days Attended 


K-2 


106 


122 


+16 


3-5 


100 


118 


+18 


6-8 


70 


93 


+23 


Overall Average 


94 


115 


+21 



Exhibit reads: On average, youth in grades K-2 attended the program 106 days in 2005-06, compared with 122 days 
in 2007-08. This change represented an average increase of 16 days. 
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Attendance Rate 



On average, students attended 8 1 percent of the days that were possible for them to attend 
in 2007-08. The attendance rate was high and consistent across grade spans, as shown in Exhibit 
5. It was also higher than the rates for participants in the CORAL programs (73 percent) and for 
elementary-grades (37 percent) and middle-grades (18 percent) participants in 21 st Century 
afterschool programs (James-Burdumy et ah, 2005). 

Exhibit 5 

Attendance Rate, by Grade, 2005-06 and 2007-08 



Grade 


Average Attendance 
Rate in 2005-06 


Average Attendance 
Rate in 2007-08 


Percentage-Point Change 


Effect Size 


K-2 


78 


84 


+6 


0.30 


3-5 


76 


82 


+6 


0.30 


6-8 


62 


71 


+9 


0.34 


Overall Average 


73 


81 


+8 


0.35 



Exhibit reads: On average, youth in grades K-2 attended the program 78 percent of the days possible in 2005-06, 
compared to 84 percent in 2007-08. This change represented a six percentage-point increase from 2005-06 and a 
medium effect size of 0.30. 



Interview and survey data from the executive directors, site coordinators, and youth all 
suggest several possible explanations for this relatively high attendance rate. For example, most 
of the executive directors (52 percent) reported in 2007-08 that, compared to other afterschool or 
out-of-school time programs that their organizations operated, their New Jersey After 3 programs 
enforced minimum attendance policies for participants “somewhat more or much more.” 

Several site coordinators attributed the high attendance rate to factors such as the types of 
afterschool activities offered, the absence of other youth programs in local communities, and the 
needs of working parents. This latter explanation was reinforced by parents, many of whom told 
us during interviews that they were able to work or to pursue employment opportunities only 
because they had somewhere safe for their children to go after school. 

Attendance may also have been high because, as a majority of youth indicated, they were 
satisfied with what the program offered. For example, 76 percent of all participants in 2007-08 
(71 percent of students in grades 3-5 and 87 percent of those in grades 6-8) “agreed” or “agreed a 
lot” that the afterschool program is a good place to spend time. The difference between the 
younger and older students was significant (effect size=0. 17). The reason for this difference is 
unclear. It may reflect the fact that older youth are often given more choice in selecting the 
afterschool program activities in which they will engage, or (as we were told by several staff 
members during visits) that programming for youth in middle school may emphasize sports and 
recreation rather than academics in order to recruit and retain youth in the older group. 
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Program Effectiveness in Retaining Youth in Programs 



We examined the rates at which NJ After 3 participants from 2006-07 re-enrolled in the 
same NJ After 3 program in 2007-08, a rate we term the “retention rate.” These analyses 
accounted for youth who would have aged out because they reached the highest grade served in 
their program. 3 For example, fifth- graders participating in programs that only served grades K-5 
in 2006-07 would be excluded from analyses because they would be ineligible to return to the 
same program as sixth-graders in the following year. This analysis does not account for family 
mobility. Even so, retention analysis is an important window on program quality because youth 
will tend to re-enroll in programs that are meeting their needs and have been beneficial to them 
in the past. 

Of the 12,104 participants who attended a New Jersey After 3 program in 2006-07 and 
were eligible to return to the same program in 2007-08, 5,567 (46 percent) re-enrolled in the 
same NJ After 3 program. Although this represented a decline from the previous period (from 
2005-06 to 2006-07) when the retention rate was 5 1 percent, the overall retention rate for 2007- 
OS exceeded the retention level of DYCD-OST programs (36 percent) from 2005-06 to 2006-07. 
The retention level of all New Jersey After 3 programs is summarized in Exhibit 6, using three 
bands to depict low, medium, and high retention. 



Exhibit 6 

Participant Retention in Programs across Two Years 



Retention Level 


Programs at This Level 
from 2005-06 to 2006-07 


Programs at This Level 
from 2006-07 to 2007-08 


Low (33% or less) 


1 1 (1 9%) 


17 (20%) 


Medium (34% to 65%) 


33 (58%) 


59 (68%) 


High (66% to 99%) 


13 (23%) 


11 (12%) 


Total 


57 (100%) 


87(100%) 



Exhibit reads: Eleven NJ After 3 programs (19 percent) had “low” retention levels from 2005-06 to 2006-07. 
Seventeen programs (20 percent) had “low” retention levels from 2006-07 to 2007-08. 



Most programs retained less than 65 percent of eligible participants from 2005-06 to 
2006-07 and from 2006-07 to 2007-08. Our interview data suggest that possible reasons for this 
finding include family mobility, competition from other programs that focus on a narrower range 
of youth needs (e.g., tutoring programs, sports teams), and parents’ inability to pay afterschool 
program fees that have been recently introduced or increased at some sites. Approximately one- 
third of site coordinators (36 percent) reported that they charge a fee. A more definitive 
explanation will require additional study, although the available data revealed an interesting 
relationship between program size and retention rate, as discussed below. 



3 The retention rate is the number of participants who returned to a NJ After 3 program in 2007-08 divided by the 
number of participants eligible to return (i.e., participants who attended a program serving their grade level). Youth 
who attended in 2006-07 but not in 2007-08, after accounting for those who would have aged out of the program, 
were considered non-returners. For all programs, evaluators empirically determined the highest grade level served 
by examining the grade levels of students served by the programs in the previous year (2006-07). 
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Programs with low retention rates tended to experience larger declines in enrollment from 
2006-07 to 2007-08, as seen in Exhibit 7. The cause of the enrollment decline is unclear, but 
several site coordinators during our visits reported that financial constraints and limited staff 
capacity affected their programs’ ability to serve youth. If programs had to reduce their program 
size in 2007-08 for these reasons, then they would also limit the number of eligible students from 
2006-07 who could return, thus lowering retention rates. 



Exhibit 7 

Change in Program Size from 2006-07 to 2007-08 and Retention Rate 




Retention Rate (Percent) 



Exhibit reads: NJ After 3 programs that had a retention rate of 50 percent or lower tended also to 
experience decreases in program size from 2006-07 to 2007-08. 
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3. Youth Development Outcomes 



Better attitudes toward school and engagement in learning are two positive outcomes that 
are commonly associated with promising afterschool programs (Little, Wimer, & Weiss, 2008). 

As the findings described in this section indicate, participants in NJ After 3 programs display 
consistent evidence of positive attitudes toward schooling and engagement in learning. In fact, 
program staff, classroom teachers, parents, and youth identified a wide range of social, personal, 
attitudinal, behavioral, and academic outcomes among youth who participated in NJ After 3. This 
section identifies and discusses these outcomes as well as significant outcome changes between the 
first and third years of the evaluation. 



Social and Personal Outcomes 

During site visits, site coordinators and program staff noted that the behavior of youth in 
the program often improved over time, and several cited examples of increased confidence and 
engagement among participants. Although cautious about attributing these improvements to the 
program exclusively, staff generally said that the program’s positive environment, coupled with 
opportunities for youth to engage in new and interesting activities, contributed to positive youth 
outcomes. For example, one staff member noted that opportunities to interact with peers 
contribute to “better manners and respect” among youth, while another stated that an emphasis on 
hands-on activities rather than traditional instruction at his program had helped “kids believe they 
can make things happen.” Others emphasized that non-academic outcomes are extremely 
important and also more likely to emerge among participants than are academic outcomes. “I 
might not see a kid go from a C to an A,” explained one site coordinator, “but I might see the 
indirect effect. . .confidence to try a math problem or to raise a hand [in response to a question in 
class] or to engage in public speaking.” 



Teacher Reports of Youth Outcomes 

Through surveys, teachers provided feedback on participants’ skills necessary for academic 
competence, including interpersonal skills, classroom engagement, motivation, and study skills. 
(These skills are also considered to be essential skills for workplace and civic success in the 21 st 
Century.) Teachers based their assessments on the skills and behaviors that afterschool 
participants exhibited during the regular school day. The findings described here are based on 
surveys of reading/language arts teachers of NJ After 3 participants who were in grades 5-8 during 
2007-08 and who were also attending one of the 10 NJ After 3 programs in the in-depth sample. 

In 2007-08, teachers reported that 68 to 75 percent of afterschool participants almost always 
or often demonstrated each of the 10 interpersonal skills identified in the evaluation’s survey (e.g., 
follows classroom rules, accepts suggestions from teachers, interacts appropriately with adults, works 
effectively in small groups) and that 75 percent almost always or often accepted suggestions from 
teachers. Teachers also reported that approximately 60 percent of participants demonstrated six of 
eight listed behaviors that relate to academic engagement (e.g., asks questions about tests or projects, 
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speaks in class when called upon, volunteers answers to questions) and that 43 to 62 percent of 
participants demonstrated each of the 1 1 behaviors and attitudes that relate to motivation in the 
classroom (e.g., persists when task is difficult, critically evaluates own work, is goal oriented, stays 
on task). Half of all students (50 percent) were reportedly almost always or often goal-oriented. 



Youth Reports of Outcomes 

Survey data indicate that participants in NJ After 3 describe their attachment to the NJ 
After 3 program in positive terms. This reaction is important because it indicates participants’ 
likely receptivity to new learning opportunities. For example, as shown in Exhibit 8, a large 
majority of participating youth reported in 2007-08 that they agree a lot or agree that they are 
successful (86 percent), they matter (81 percent), and their ideas count (73 percent). 

Exhibit 8 

Participants’ Self-reported Level of Attachment to the Program, 
Grades 3-8, in Percents, 2005-06 and 2007-08 



In this afterschool program, I feel like... 



I am safe 



I am successful 



I belong 



I matter 



This is a good place to hang 
out 



My ideas count 

0 20 40 60 80 100 

Percent of participants who reported they "agree" or "agree a lot" 

■ 2005-06 (n=627) □ 2007-08 (n=916) 



Exhibit reads: In 2007-08, 88 percent of participants agreed or agreed a lot that they feel safe in their NJ After 3 
program, compared with 91 percent of participants in 2005-06. This difference is not statistically significant. 

In addition, as illustrated in Exhibit 9, participants expressed very positive attitudes toward 
staff in their afterschool program. More than 80 percent of respondents said they believed that staff 
think they can do things well (84 percent), treat them with respect (84 percent), try to be fair (83 
percent), think they can learn new things (82 percent), and really care about them (8 1 percent). 
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Exhibit 9 

Youth Perceptions About Staff, Grades 3-8, in Percents, 2005-06 and 2007-08 



In the afterschool program... 

Staff think I can do things w ell 
Staff treat me w ith respect 
Staff try to be fair 
Staff think I can learn new things 
Staff really care about me 
Staff help me to try new things 
Staff care w hat I think 

Staff often keep their promises 

I feel that I can talk about things that 
are bothering me 

0 20 40 60 80 100 

Percent of participants who reported they "agree " or "agree a lot" 

— 

■ 2005-06 (n=633) 0 ) 2007-08 (n=927) 

Exhibit reads: In 2007-08, 84 percent of participants agreed or agreed a lot that staff think they can do things well in 
their afterschool program, compared to 88 percent in 2005-06. The difference is not statistically significant. 




Site coordinators recognized the importance of recruiting qualified staff and of maintaining 
a roster of well-trained, engaged, and creative individuals on staff who can relate well to 
participants and provide the encouragement and guidance they need. According to site 
coordinators, staff members who possess diverse skills and abilities are particularly valuable 
because they help to create the atmosphere and tone that attract youth and that facilitate youth 
growth and development. Many expressed appreciation for those who work at their program who 
have demonstrated these abilities. “I am confident in my staff to do academic work with the kids,” 
explained one site coordinator. “They are good [and] most have been here for a while. A majority 
of them are going to become teachers... they are good at the [teaching and] learning component.” 
Others emphasized the excellent relationships between their staff and youth, noting that some staff 
members have become mentors who, as one person explained, “show the kids how to be 
successful.” Positive comments regarding staff quality and their ability to provide instruction and 
to build relationships included the following: 



1 think that [my staff’s [...strength is their ability to engage and develop relationships with 
the children. I think that [with children] at this age it is really about relating to [them]. 
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[My] instructional staff are the ones who encourage the students [and] the youth 
developers are [the ones who] develop the relationships. [They] get the students to buy 
into participating and [help] keep them here. The instructors are there to provide the 
academic support. 

Participants also indicated that they enjoy positive relationships with peers in the 
program. As illustrated in Exhibit 10, a majority reported in 2007-08 that, for example, they had 
a lot of friends in the program (89 percent), they have a good time playing with other youth in 
the program (85 percent), and, in the program, they get to work with other youth as part of a 
team (83 percent). 



Exhibit 10 

Youth Perceptions About Peer Relationships, Grades 3-8, 
in Percents, 2005-06 and 2007-08 



In the afterschool program,!... 



Have a lot of friends 




■ 89 




Get to know other kids really 




| 86 


well 




Have a good time playing w ith 




| 85 


other kids 




Get to w ork w ith other kids as 




~]83 


part of a team 




Get along w ith other kids 




| 81 




Like the other kids 




| 78 




Can really trust the other kids 




■ 62 






, , , , 



0 20 40 60 80 100 



Percent of participants who reported they "agree " or "agree a lot" 



■ 2005-06 (n=630) □ 2007-08 (n=920) 



Exhibit reads: In 2007-08, 89 percent of participants agreed or agreed a lot that they have a lot of friends in their 
afterschool program, compared to 88 percent in 2005-06. The difference is not statistically significant. 

These positive results were also found in similar evaluations of afterschool programs in 
New York and Los Angeles. For example, LA’s BEST evaluators noted that one of the most 
significant findings of that study was that, overwhelmingly, students enjoyed the program. A 
majority of participants reported that it was fun and safe, and they wanted to keep going back. 
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More than half of those who participated in TASC programs in New York City also indicated a 
strong sense of connections in their program. 

Responses by NJ After 3 participants regarding their relationships with staff and with 
peers were also similar to those reported by youth in the DYCD-OST programs. Four-point 
scales were created in both studies to measure responses to statements related to staff interactions 
and to peer relationships. The mean scores on the scale were 3.23 for staff interactions and 3.27 
for peer relationships for NJ After 3 participants. The mean scores for participants in DYCD- 
OST programs were similar — 3.34 for staff interactions and 3.27 for peer relationships. 

In addition, many youth reported that they engage in positive social interactions with 
others. For example, a majority of those surveyed reported in 2007-08, as shown in Exhibit 11, 
that, on at least three occasions in the past 30 days, they had given someone a compliment (66 
percent), helped another student solve a problem (61 percent), or cooperated with others in 

Exhibit 11 

Participant Self-reports of Positive Social Behaviors, Grades 3-8, 
in Percents, 2005-06 and 2007-08 



In the past 30 days I... 



Gave someone a compliment 
Helped other students solve a problem 
Cooperated with others in completing a task 

Told other students how I felt when they upset me 

Told other students how I felt when they did something I 
liked 

Protected someone from a bully 
Helped someone stay out of a fight 



0 20 40 60 80 100 

Percent of participants who reported doing the activity at least three times 



■ 2005-06 (n=629) □ 2007-08 (n=923) 



Exhibit reads: In both the first and third years of the evaluation, 66 percent of participants said that 
they gave someone a compliment at least three times in the past 30 days. 
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completing a task (60 percent). Many also reported that, at least three times during the past 30 
days, they had told other students how they felt when upset due to actions of the other student 
(49 percent), told other students how they felt when others did something they liked (48 percent), 
protected someone from a bully (40 percent), or helped someone stay out of a fight (38 percent). 
Although a causal relationship between participation in NJ After 3 and these positive behaviors 
cannot be established from available data, program staff said that character education programs 
and activities intended to develop these behaviors were offered at their sites. 

Results from the participant survey also suggest that leadership skills and a sense of civic 
responsibility have been cultivated among many NJ After 3 participants, as shown in Exhibit 12 
and Exhibit 13, respectively. About half of all respondents indicated in 2007-08 that they had 

Exhibit 12 

Participant Self-reports of Youth Leadership Opportunities, Grades 3-8, 

in Percents, 2005-06 and 2007-08 



At the afterschool program I have... 



Helped plan a program activity or event 



Been asked by staff for my ideas about the program or an 
activity 



Led an activity 



Helped out on a youth council, advisory group, or leadership 
team for this program 



Helped plan a commuity service project 



Helped out in the office 



Helped with meetings for parents or community member! 




0 20 40 60 80 

Percent of participants who reported doing the activity 



I 2005-06 (n=650) 



□ 2007-08 (n=927) 



100 



Exhibit reads: In 2007-08, 56 percent of participants reported that they helped plan a program activity or 
event in their NJ After 3 program, compared with 60 percent of participants in 2005-06. The difference is not 
statistically significant. 
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Exhibit 13 

Participant Self-reports of Community Service Opportunities, Grades 3-8, 

in Percents, 2005-06 and 2007-08 



Participating in community service projects 
with this afterschool program, I feel like ... 

It is important to volunteer and to help others 
I w ill continue to volunteer to help others in my community 

My actions help others 
I have learned more about how I can help others 
I can call myself a volunteer 
I have learned more about my community 

My actions make a difference in the community 

I have learned more about other organizations in my 
community 




0 20 40 60 80 1 00 



Percent of participants who reported they "agree" or "agree a lot" 



I 2005-06 (n=630) 



□ 2007-08 (n=889) 



Exhibit reads: In 2007-08, 83 percent of participants agreed or agreed a lot that it was important to volunteer 
and to help others, compared to 82 percent in 2005-06. The difference is not statistically significant. 



helped plan a program activity or event (56 percent), led an activity (49 percent), or been asked 
by staff to share ideas about the program or an activity (49 percent). In addition, when asked 
about their experiences as a community project volunteer, a majority (83 percent) reported that 
they agreed or agreed a lot that it is important to volunteer and help others. A majority also 
reported that they agreed a lot or agreed with several other statements that together reflect a 
civic-minded attitude (e.g., “My actions make a difference in the community,” “I will continue to 
volunteer and help others in my community”). 



Comparing NJ After 3 participants to participants in other programs, a majority of youth 
in both NJ After 3 and in the DYCD-OST programs agreed a lot or agreed that they are 
successful, they belong, and they matter. LA’s BEST evaluators noted that one of the most 
significant findings of their study was that, overwhelmingly, students enjoyed the program, 
thought it was fun and safe, and wanted to keep going back. 
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NJ After 3 has adopted healthy development among children as one of its goals. While no 
statistical claims can be made regarding the relative effectiveness of NJ After 3 programs in 
addressing this issue, the survey self-reports suggest that there was little change in health 
characteristics among participants between the first and third years of the evaluation. The 
percentage of participants who reported that they engage in physical activity, watch television, play 
video games, and eat certain foods for specified durations or with certain frequencies each week 
remained about the same. In 2007-08, 28 percent of youth reported that they participated in dance 
for three or more hours each week, and 41 percent reported that they played basketball, football, or 
soccer for three hours or more each week. Forty-one percent said they watch television and 23 
percent said they play video games on a typical school day for four or more hours. 

In addition to participation in physical activities, participants answered questions about 
their nutrition and sleeping habits. Here again, their responses suggested little change over the 
three years. In 2007-08, 22 percent of respondents indicated that they eat food from a fast food 
restaurant three or more times per week, and 73 percent of respondents said that they sleep eight 
or more hours on a typical school night. 



Parent Reports of Youth Outcomes 

In general, parents were satisfied and happy with their children’s NJ After 3 experiences 
and outcomes. When asked about the types of activities in which their children participated, a 
parent said, “They wrote their own play. . .they wrote their own songs. If they weren’t coming 
here they wouldn’t have been exposed to this. He’s [learning how to] be responsible. . .I’m very 
happy. They give them a lot of opportunities to do a lot of things.” “My child is currently 
involved in cooking, technology, and reading. She does a lot of the sports program. It helps 
them stay active. I think it helps because she gets to socialize with her friends,” said another. 
“Everything is organized. He has a schedule of what they do. They have all the different 
programs [such as] art and literature. He writes stories — he was so impressed because the 
teacher had so many good comments. It encourages him,” said a third. 

Parents also described the ways in which their children had changed as a result of their 
participation in the NJ After 3 program. “Socially, she’s interacting with other people and opening 
up a lot more,” explained one parent. Other parents talked about how their child’s academic skills 
improved. “He learns from doing his homework. They’re good at helping him with the homework, 
so he leams better about what he is working on in school,” said one parent. Additional parents noted 
that their children’s language skills had improved. “[The program] helps with [learning] new 
vocabulary. The English has helped them a lot, since I’m an immigrant,” explained one parent. 



Academic Outcomes 

NJ After 3 is not designed solely as an academic program. Rather, it is a comprehensive 
afterschool program that incorporates academics, public safety, youth development, and workforce 
development. Because of understandable interest in participants’ academic development, the 
evaluation included measures in this domain. 
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This section identifies academic outcomes among participating youth that were reported by 
teachers and by youth who responded to survey and interview questions regarding their personal 
academic achievements and improvements. During interviews, site coordinators and other 
program staff discussed, in general terms, the ways in which afterschool activities encourage 
positive youth development and academic achievement. “The [important] thing,” explained one 
staff member, “is [we are] there for them. ..giving support. Sometimes it’s not that [youth] don’t 
know the stuff or can’t do it, but they just need to be encouraged.” Others emphasized that 
compliments, gentle prodding, and clear expectations are effective means by which afterschool 
program staff have succeeded in motivating students to try harder in school and ultimately improve 
academically. For example, one staff member reported that youth in her program have “fallen into 
line in tenns of doing homework” because it is a required of all participants in the afterschool 
program. “Homework is about 25 or 30 percent of [a student’s] grade,” she explained, “so that has 
really helped student grades.” 



Teacher Reports of Youth Outcomes 

Teachers identified important academic outcomes among students who participated in NJ 
After 3 in academic engagement and interpersonal skills, study habits, reading and language arts, 
and technology skills during each year of the evaluation. As shown in Exhibit 14, teachers 

Exhibit 14 

Teacher Reports of Participants’ Academic Engagement and Interpersonal Skills, 

Grades 5-8, in Percents, 2007-08 

Speaks in class when called upon 
Gets alo ng with people who are d ifferent 
Followsclassroom rules 
Accepts suggestions from teachers 
Interacts appropriately with adults 
Co r rects i n appropri ate behav ior wh en as ked 
Works effectively in a small group activity 
Listens to whatothers have to say 
Interacts appropriately with other students 
Works effecti vely i n a I arg e g roup activ ity 
Expresses dissatisfaction appropriately 

65 70 75 80 85 

Percent of participants as reported by teachers 
(n=209) 



Exhibit reads: According to teachers, 77 percent of participants in grades 5-8 “almost always” or “often” speak in 
class when called upon. 
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reported in 2007-08 that 77 percent of students who participated in NJ After 3 “almost always” 
or “often” speak in class when called upon and that 75 percent get along with people who are 
different, follow classroom rules, and accept suggestions from teachers. 

Similarly, as shown in Exhibit 15, teachers reported that over 70 percent of students 
“almost always” or “often” take care of materials, complete homework, and turn in homework on 
time. 



Exhibit 15 

Teacher Reports of Participants’ Study Skills, Grades 5-8, in Percents, 2007-08 



Takes care of materials 
Co mp letes h omework 
Turns in homework on time 
Finishes class work on time 
Co mp I etes assignments according to d irections 
Prepares for class 
Pays attention in class 
Prepares for tests 
Takes notes in class 
Reviews materials 
Corrects own work 

0 20 40 60 80 100 

Percent of participants as reported by teachers 

(n=207) 



Exhibit reads: According to teachers, 74 percent of participants in grades 5-8 “almost always” or “often” took care of 
materials. 




According to teachers, the percentage of students who met or exceeded grade-level 
expectations in several key reading and language arts areas was high. As shown in Exhibit 16, 
75 percent or more met or exceeded grade-level expectations in punctuation, drawing 
conclusions from written materials, spelling, identifying main ideas, and oral communication. 
Teachers also reported a significant difference between older and younger students in 2007-08; 
83 percent of students in grades 7-8 were above grade level in reading fluency, compared to 69 
percent of students in grades 5-6 (effect size=0.15). 
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Exhibit 16 

Teacher Reports of Participants’ Reading and Language Arts Skills, 
Grades 5-8, in Percents, 2007-08 



Oral communication 
Identifying a main idea 
Spelling 

Drawing conclusions from written material 
Punctuation 
Read i n g comprehension 
Vocabulary 
Written communication 
Reading fluency 
Word attack 
Grammar 




Percent of participants as reported by teachers 

(n=209) 



Exhibit reads: According to teachers, 83 percent of participants in grades 5-8 met at least grade-level expectations for 
oral communication skills. 



Teachers also reported important technological skills among a majority of students. As 
shown in Exhibit 17, 77 percent of students were excellent or good at using the Internet, 75 
percent of students were excellent or good in using word processing programs, and 65 percent of 
students were excellent or good at using spreadsheet programs. 

Analyses to determine the extent to which teachers’ reports of NJ After 3 participants 
improved from 2006-07 to 2007-08 revealed a positive trend. However, results did not meet the 
evaluation’s significance threshold. One notable exception is teachers’ assessment of 
participants’ technology skills, which improved significantly from 2006-07 to 2007-08. For 
example, in 2007-08, teachers reported that 81 percent of students exhibited excellent or good 
technology skills when playing games, compared to 71 percent in 2006-07 (effect size=0.1 1). In 
2007-08, teachers reported that 77 percent of students exhibited excellent or good skills when 
they used the internet for research, compared to 61 percent in 2006-07 (effect size=0.17). 
Similarly, 75 percent of participants in 2007-08 exhibited excellent or good skills when using a 
word processing program, compared to 56 percent of participants in 2006-07 (effect size=0.19). 
Seventy-four percent of participants in 2007-08 exhibited excellent or good skills when they sent 
and received e-mail, compared to 59 percent in 2006-07 (effect size=0.16). Lastly, teachers 
reported that the percentage of participants who exhibited excellent or good skills when using a 
spreadsheet program also increased between the two periods from 45 percent in 2006-07 to 65 
percent in 2007-08 (effect size=0.20). 
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Exhibit 17 

Teacher Reports of Participants’ Technology Skills, Grades 5-8, in Percents, 2007-08 



Playing games 



Useofthe Internet for 
research 



Useofword processing 
program 



Sen d i ng an d receivi ng emai I 



Use of spreadsheet program 




0 20 40 60 80 100 



Percent of participants as reported by teachers 

(n =1 76) 



Exhibit reads: According to teachers, 81 percent of participants in grades 5-8 exhibited “excellent” or “good” 
technology skills in playing games. 



The reader should use caution when drawing conclusions from these findings. The 
preceding analyses compare participants in grades 5-8 in 2007-08 with participants in grades 4-7 
in 2006-07. It would be expected that participants’ skills in these areas would increase because 
they are older and are likely to be using technology more often. 

We also used teacher survey items to create a five-point scale that examined teacher 
ratings of participants’ reading and language arts skills in comparison with the grade-level 
expectations in their school. A score of 5 indicates that, on average, participants’ skills were far 
above grade-level expectations, and a score of 1 indicates that the rated skill is far below 
average. Across all grade levels, teachers recorded an overall mean scale score of 2.97 on their 
assessment of participants’ reading and language arts skills in 2007-08. 

To assess student growth over time, we compared reading and language arts scale items 
for participants who had been assessed by their teacher in two consecutive years of the 
evaluation. This is a more accurate measure of growth than an examination of all teacher reports 
in one year and all teacher reports in a subsequent year. By focusing on the same students in 
consecutive years, our analysis examines data for those youth who received sustained exposure 
to the program and who were therefore most likely to have benefited from participation. A total 
of 86 students in the in-depth sites satisfied these criteria, with teacher reports in two consecutive 
years. 
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We compared each student’s scale scores in the first year (which could have been 
recorded in either the first and second year of the evaluation) with his or her scale scores in the 
second year (which could have been recorded in either the second or third year of the 
evaluation). Analysis showed that the mean scale score for participants in the first year was 
2.79, and the mean score for participants in the second year was 3.14, representing a significant 
gain from the earlier to later measure (effect size=0.24). 

While these data suggest a significant improvement, most of the increase was evident 
among a subset of the 86 students. Calculations to identify any statistical associations between 
the scale-score change and attendance rates or days enrolled in the program did not yield a 
correlation. In addition, it is possible that student self-selection contributed to the observed 
pattern. Thus, we cannot be certain that participation in NJ After 3 contributed to the gain. 
Nevertheless, this result is noteworthy and warrants further study. 

In addition to the reading and language arts scale, we created a study skills scale that 
included, for example, teachers’ assessments of how well or how often students prepared for 
tests, took notes in class, completed homework, corrected their own work, and paid attention in 
class (as listed in Exhibit 15). Participants who had been identified as “highly active” in the 
program (those who attended at least 80 days and attended at least 80 percent of the days they 
were enrolled) were rated on the study-skills scale at 3.8 on a five-point scale. This score was 
significantly higher than the mean scale sore of 3.5 that was received by “non-active” 
participants (those who attended fewer than 60 days or less than 60 percent of the time for which 
they were enrolled) (effect size=0.15). This relationship between NJ After 3 participation level 
and teacher-rated study skills suggests the possibility that learning experiences in the NJ After 3 
program positively influenced students’ study skills, although it is also possible that student self- 
selection accounted for some or all of the difference. 



Youth Reports of Educational Growth 

As illustrated in Exhibit 18, participants believe that NJ After 3 has helped them 
academically. In 2007-08, 84 percent of youth respondents reported that they agreed or agreed a 
lot that the program helped them to finish their homework more often, 73 percent reported that 
the program helped them get better grades in school, 69 percent indicated that the program 
helped them to read and understand better, 67 percent said it helped them solve math problems 
better, 62 percent said it helped them write better, and 5 1 percent reported that the program 
helped them to use computers to do schoolwork better. Older students in particular reported that 
they had benefited from increased access to technology in the afterschool program. Sixty 
percent of students in grades 6-8, compared to 46 percent in grades 3-5, agreed or agreed a lot 
that their afterschool program has helped them use computers to do schoolwork better (effect 
size=0.13). 
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Exhibit 18 

Youth Perceptions of the Program’s Help with School, Grades 3-8, 
in Percents, 2005-06 and 2007-08 



This afterschool program has helped me to... 



Finish my homework more often 



Get better grades in school 

Feel better about my 
schoolwork 



Read and understand better 



Solve math problems better 



Write better 

Use computers to do 
schoolwork better 



0 20 40 60 80 100 

Percent of participants who reported they "agree " or "agree a lot" 



■ 2005-06 (n=642) O 2007-08 (n=930) 



Exhibit reads: In 2007-08, 84 percent of participants agreed or agreed a lot that their NJ After 3 program helped 
them finish their homework more often, compared with 86 percent of participants in 2005-06. The difference was 
not statistically significant. 

Although youth participants were positive in their assessment of program benefits in 2007- 
OS, they were generally a bit less positive than they had been in the first year of the evaluation. 

For example, on the four-point scale created to assess participants’ views on the academic benefits 
associated with participating in NJ After 3, the mean in 2007-08 was 2.92, which was significantly 
lower than the mean of 3.03 in 2005-06 (effect size=0. 13). Nevertheless, the results are 
comparable to the findings from the DYCD-OST initiative in New York City (Russell et al., 2008). 
NJ After 3 program participants reported similar ratings to those of DYCD-OST participants on 
the series of questions about perceptions of the program’s help with school. For both of these 
evaluations, scales were created to measure the extent to which participants agreed with a series of 
statements related to getting better grades, feeling better about schoolwork, reading and 
understanding better, solving math problems better, finishing homework more often, writing better, 
and using computers to do schoolwork better. The mean score on a scale from one to four was 
2.93 for NJ After 3 participants in 2006-07, and the mean score for the somewhat older 
participants (some of whom were in high school) in the DYCD-OST programs in the same year of 
that evaluation was 2.99. 
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Youth responses were less positive in the third year of the evaluation compared to the 
first year regarding opportunities for exposure to new opportunities provided by the program. 
Ratings on the four-point scale declined from a mean scale score of 3.01 in the first year to a 
mean scale score of 2.92 in the third year (effect size=0. 13). Again, the reason is unclear. A 
possibility is that, as the interview data suggested, not all programs succeeded in providing as 
wide a range of popular youth activities (e.g., field trips, cultural activities, sports and recreation) 
as participating youth desired. The responses of NJ After 3 participants to questions about their 
exposure to new opportunities were lower than that of participants in the DYCD-OST programs 
in New York City. For both of these evaluations, scales were created to measure responses to a 
series of statements related to exposure to new experiences. The mean score on the four-point 
scale was 2.92 for NJ After 3 participants, and for participants in the DYCD-OST programs it 
was 3.17. 

We also analyzed survey data for those youth who had participated in the program for all 
three years and compared these results with results reported by youth who had participated in the 
program for only a year. The mean score on the academic benefits scale among those who had 
participated for only one year was 3.03, and the mean scale score for those who had participated 
for three years was 2.60 (effect size=0.58). 
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4. Program Content and Features 



NJ After 3 has continued to offer a wide array of academic, artistic, social, health, civic, 
and athletic activities that address the diverse needs and interests of participants. This section 
reviews site-level program activities for youth and describes how information related to these 
activities is communicated to their parents. 



Program Activities 

Site-level activities reflect the range of programmatic goals and objectives by site 
coordinators, as presented in Exhibit 19. 



Exhibit 19 

Major Program Objectives and Goals Reported by Site Coordinators, 
in Percents, 2005-06 and 2007-08 



Major Programming Objectives and Goals 


2005-06 

(n=55) 


2007-08 

(n=98) 


Provide a safe environment for youth 


100 


100 


Help youth develop socially 


100 


97 


Provide youth with positive adult guidance and/or mentors 


100 


96 


Provide hands-on academic enrichment activities 


96 


91 


Help youth improve their academic performance (e.g., grades, test 
scores) 


96 


87 


Promote respect for diversity among youth 


95 


88 


Provide physical fitness or athletic opportunities 


89 


85 


Provide recreational activities 


87 


90 


Provide opportunities for cultural enrichment 


86 


90 


Support working families 


85 


93 


Provide health/well-being/life skills development 


76 


80 


Help connect parents with their child’s school and/or community 


75 


68 


Provide community service or civic engagement opportunities 


67 


64 


Provide leadership opportunities for youth 


65 


65 


Help connect youth to their community 


53 


58 



Exhibit reads: In both years, 100 percent of site coordinators reported that providing a safe environment for 
youth was a primary goal. 



Similar program priorities were also identified by the executive directors of the CBO 
partners organizations. A majority of them also identified providing a safe environment (93 
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percent), helping participants develop socially (82 percent), helping to improve academic 
performance (75 percent), providing positive adult guidance or mentors (75 percent), providing 
opportunities for cultural enrichment (64 percent), and providing recreational activities (50 
percent), as primary objectives. The results were generally similar to responses in the 
evaluation’s first year, although a significantly larger percentage (96 percent) of executive 
directors identified providing positive adult guidance or mentors as a primary objective in the 
first year than in the third (effect size=0.27). 

Even so, as the following discussion highlights, participants’ access to certain types of 
activities varied depending on which program they attended. 



Program Variations and Changes 

We found several significant differences between programs that were led by one of the 55 
respondents who said that they had also been site coordinators during the previous (2006-07) 
year (“veterans”), compared to the 40 respondents who had not been a site coordinator in 2006- 
07. For example, a significantly larger percentage of veterans reported that they always offered: 
math games and activities (94 percent of veterans, compared to 76 percent of new site 
coordinators; effect size= 0.26); drama or theater instruction and activities (58 percent of 
veterans, compared to 17 percent of new site coordinators; effect size= 0.40); and activities to 
learn about or explore different cultures and languages (85 percent of veterans, compared to 52 
percent of new site coordinators; effect size= 0.35). Similarly, veteran site coordinators were 
significantly more likely to report that youth participated in: math games and activities (94 
percent, compared to 59 percent; effect size= 0.44); learning games and activities (96 percent, 
compared to 79 percent; effect size= 0.28); and drama and theater activities (77 percent, 
compared to 29 percent; effect size= 0.55) for at least one hour per week. This pattern was 
reversed in one content area: significantly more of the new site coordinators (94 percent, 
compared to 74 percent of the veterans; effect size= 0.26) reported that all or most youth 
participated in peer discussions of topics that are important to youth for at least one hour a week. 

As reflected in the staff comments below, responses to the questions about program 
changes typically included a discussion of improvements in organizational structures, increased 
freedom of choice for participants, and efforts to expand program offerings. 

We all have specific classes that we have to teach, specific topics and grades. [The 
program] has become more organized academically than [before]. 

I have seen growth in the program... a lot of [youth] have decided to come based on what 
their friends have said about it. [They like ] the trips that we have gone on. [There] is a 
good [number] of choices now. 

[The program] is more organized [and] there are more clubs and more activities. We 
have cooking now... [and] we have art. We have girls in one group and boys in another 
group [for some character-building ] activities. 
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The priority for us is to develop more structure [and] also more program offerings. We 
are shifting toward a focus that will make [the program] more literacy based. Kids at 
this school aren ’t on grade level in reading, so we will be grouping them for literacy 
activities. That will happen next year. 

This year [we focused] more on programming and on increasing what we offer. We want 
to offer more... This year we had to add fees for attendance, which made it even more 
important to do something differently to keep enrollment up. So we added some more 
hands-on and fun activities. We wanted to give [youth] some different types of learning 
experiences that were different from what they were used to in school. We saw the need 
and saw that the children wanted this change. It was informed feedback we got by just 
watching and talking with the kids. 

Other staff emphasized the challenges associated with trying to ensure that participants’ 
multiple needs and interests — academic, enrichment, and recreational — were all addressed. 

Their responses indicated that, while some site coordinators sought to increase academic 
offerings, others wanted to expand other types of program offerings. 

[We need] more hands-on activities [and] greater student involvement. 

I think that the [school] administration thinks that [our program] is supposed to be 
academic, and they are little disappointed [that it is not]. But if we did that, we would 
not have students sign up willingly. They will not sign up for an academic program. 
[Students] would like to pick and choose [activities]. There are some things that they are 
not very interested in, and there are [other activities] that they would like to see in place. 
[Students] would like to see more variety, but we have not been able to fully do that. We 
do need some more variety in [our] programming. 

I added more of an educational curriculum [this year]. [There is] more structure, more 
math, reading, [and] phonics. The kids say: “I’m in school cdl day, do I have to do 
this ? ” [But] I am finding out from talking to teachers [ that] a lot of kids can ’t add two 
numbers. Third-graders don ’t know their multiplication tables. We have to [ include] 
more [academic] depth. 

Our programming needs to be student based. Their input must drive what we do. So, we 
talk with them to see what they like and don ’t like and try to keep up. We do surveys. All 
that is to keep up with the kids and their changing needs. Also we must consider the 
school and be responsive to what is happening in the school. For instance, [this school] 
may lose [its] recreation funding next year. If they do, then we will have to offer more 
recreation [in the afterschool program] or else the kids will recdly not have opportunities 
for exercise and learning sports. They need that. 

Academic Offerings 

While individual sites varied in terms of the content and duration of the academic activities 
offered, everyone with whom we spoke acknowledged the importance of addressing participants’ 
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academic needs during the afterschool period. In fact, half of all executive directors reported that, 
compared to the other afterschool or out-of school time programs that their organizations operate, 
the NJ After 3 program focuses much more or somewhat more on academics. 

As illustrated in Exhibit 20, in addition to homework help, many sites offered tutoring or 
enrichment activities that addressed language arts and mathematics. We learned during site 
visits that several programs also address topics in social studies and science. 

Exhibit 20 

Academic Activities Offered by NJ After 3 Programs, in Percents, 

2005-06 and 2007-08 



Academic activities offered... 



Homework help 



Learning games/activities 
Independent reading 

M ath games/activities 

Organized writing activities (e.g., creative 
writing, newspapers) 

Organized reading activities 

Group instruction in specific academic 
subjects 

T raining in co mputer skills 

T raining in study skills 

Field trips to high school/college 
campuses 

0 20 40 60 80 100 

Percent of site coordinator survey respondents 



■ 2005-06 (n=55) □ 2007-08 (n=97) 

Exhibit reads: In both years, all responding site coordinators (100 percent) reported that their afterschool 
program offered homework help. 

At a few of the sites we visited, we were told that academic programming decisions are 
guided by input from teachers and administrators who work at the school during the regular 
school day. Several NJ After 3 staff members emphasized, however, that they strive to ensure 
that academic instruction is delivered in a way that differs from the school-day instruction. 

We try to be consistent with the school on concept and content. If the school is doing 
multiplication, we are as well. If they are studying fruits and vegetables, then we do too. 
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We stay consistent with the school day, but with a twist... we try to keep [students’] 
interest and to keep them happy. 



We try to have certified teachers [in the program] at the beginning of the school year to 
emphasize academics. We only keep these teachers for a [limited] period of time... we 
can 7 afford to keep them the whole year. This year we had [certified] teachers from 
October to April. If [students] are learning fractions, we make sure the teachers are 
[instructing] them in how to do fractions. After the teachers leave [in April], we still 
offer math in the afterschool program, but it’s more like fun math. There is less direct 
instruction as the year goes on. 

We have employed some certified teachers from this school and they are very involved 
and very in tune with making what we teach in afterschool close to the regular school. 
They have a test called NJ ASK, and the teachers use the afterschool time to teach that 
test. It is not teaching the test, but teaching fractions, multiplication, or whatever skills 
are required to do well on that test. There is a conscious effort to do that. 



As illustrated in Exhibit 21, a majority of site coordinators in 2007-08 reported that they 
discussed the following academic -related issues with school staff on a monthly basis: the needs 



Exhibit 21 

Program Communication with the School, in Percents, 2005-06 and 2007-08 



Discuss with school staff at least once a month... 

The needs or progress of individual students 

Issues related to classrooms/sharing space 
Homework assignments 
Student discipline policies 



Plans for afterschool program content’ 



Afterschool program staffing 



Curriculum concepts currently being taught 
in school* 



Afterschool program enrollment policies 
(e.g., targeting students)* 



State and local standards in reading, math, 
and/or science 




20 40 60 80 

Percent of site coordinator survey respondents 



100 



1 2005-06 (n=55) 



□ 2007-08 (n=97) 



Exhibit reads: In 2007-08, 79 percent of site coordinators reported that they discussed the needs or progress of 
individual students with school staff at least once a month. In 2005-06, 71 percent of site coordinators had those 
conversations at least once a month. This difference is not statistically significant. 

* Difference between years is significant at p<.05 
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or progress of individual students (79 percent), homework assignments (74 percent), and plans 
for afterschool program content (54 percent). Discussions regarding afterschool program content 
and in-school curriculum concepts were held by significantly fewer site coordinators in 2007-08 
(54 percent and 46 percent, respectively), compared with 2005-06 (79 percent and 66 percent, 
respectively; effect sizes= 0.26 and 0.30 respectively). This shift may reflect a growing 
familiarity over time between the school-day and afterschool programs, which reduces the need 
for explicit communications. 



Artistic and Social Development Activities 

As illustrated in Exhibit 22, more than three-quarters of site coordinators reported in 
2007-08 that they provided: visual arts and crafts instruction (96 percent), organized social 
events (89 percent), activities to help participants learn about different cultures (89 percent), 
dance instruction (86 percent), unstructured time for socializing (85 percent), opportunities to 

Exhibit 22 

Artistic and Social Development Activities, in Percents, 2005-06 and 2007-08 



Arts and cultural activities offered... 

Visual arts and crafts instruction/ projects (e.g., painting, drawing, pottery) 

Organized social events (e.g., parties, festivals) 
Activities to learn about/explore different cultures/languages 
Dance/movement instruction or activities 
Unstructured time for socializing 
Creative writing (e.g., poetry, short stories) 
Discussion about diversity issues 
Activities to discuss gender issues and empowerment 
Drama/theater instructio n or activities* 
M usic instruction/ activities 

Field trips to performances or exhibits 

Opportunities to meet and talk with professional artists and performers (other than 
program staff) 

Arts-related technology (e.g., video production, film-making, website development) 
Mentoring (e.g., Big Brothers/Big Sisters) 



■ 2005-06 (n=55) 0 2007-08 (n=97) 




Percent of site coordinator survey respondents 



Exhibit reads: In 2007-08, 96 percent of site coordinators reported that their afterschool program offered visual arts 
and crafts projects or instruction, compared with 100 percent of site coordinators in 2005-06. This difference is not 
statistically significant. 
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youth to engage in creative writing (84 percent), and opportunities to youth to discuss diversity 
issues (79 percent). The only significant change in the types of artistic activities offered between 
2005-06 and 2007-08 was in the percentage of site coordinators (59 percent in 2007-08, 
compared to 82 percent in 2005-06; effect size= 0.23) who reported that they offered drama 
instruction or activities in their programs. 



Civic Engagement, Community Service, and Career Exploration 

As illustrated in Exhibit 23, most site coordinators reported that they provided 
opportunities for participants to engage in civic discussions and career exploration activities. 
Only a few offered service projects and opportunities for youth to acquire work experience, 
which likely reflects the fact that most participants in NJ After 3 are elementary-age youth for 
whom such activities may be inappropriate. 

Exhibit 23 

Civic Engagement, Community Service, and Career Exploration Activities, 

in Percents, 2005-06 and 2007-08 

Civic engagement and community ctivities offered... 

Discussion of current events 

Discussion of issues, events, or problems in your community 
Career explo ratio n activities 

Service projects in the neighborhood/nearby schools (e.g., tutoring, visiting the elderly, 
block clean-up) 

Mock government or election activities or projects 
Field trips to local businesses 
Orientation to job search and basic employability skills 
Civic simulation games (e.g., SimCity, the Village Project) 

Job shadowing opportunities 
Internships or apprenticeships 
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Percent of site coordinator survey respondents 



■ 2005-06 (n=55) ■ 2007-08 (n=97) 



Exhibit reads: In 2007-08, 76 percent of site coordinators reported that their afterschool program offered discussion 
of current events as an activity, compared with 68 percent of site coordinators in 2005-06. This difference is not 
statistically significant. 
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Athletic, Health, and Life Skills Activities 



Exhibit 24 shows that a majority of site coordinators in 2007-08 offered athletic activities 
such as free time for physical play or pick-up sports (94 percent) and organized team sports (77 
percent), health and nutrition education (86 percent), peer discussion groups (76 percent), and 
conflict resolution training (68 percent). 



Exhibit 24 

Athletic, Health, and Life Skills Activities, in Percents, 2005-06 and 2007-08 



Athletic and health activities offered... 

F ree time fo r physical play o r pick-up spo rts 

Health or nutrition education 

Organized team sports instruction/ activities 

Peer discussion of topics that are important 
to youth 

Conflict resolution training 
Instruction in life skills 

Fitness classes or activities 

Organized individual sports instruction/ 
activities 

M artial arts instruction and practice 

0 20 40 60 80 100 

Percent of site coordinator survey respondents 

■ 2005-06 (n=55) n 2007-08 (n=97) 

Exhibit reads: In 2007-08, 94 percent of site coordinators reported that their afterschool program offered free time for 
physical play or pick-up sports, compared with 96 percent of site coordinators in 2005-06. This difference is not 
statistically significant. 




Calculations to determine correlation between the activities in which youth participate, 
attendance, and reported youth outcomes (through youth responses to specific survey items) did 
not yield results that met our criteria for statistical significance and effect size. Nevertheless, 
statistically significant correlations between activities and self-reported youth benefits were 
identified: 



■ The extent to which participants reported that they benefited academically by 
participating in the program was positively associated with the availability of 
community service projects, as reported by site coordinators (r=0.86, effect 
size=0.74). 
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■ The extent to which participants reported that they benefited academically by 
participating in the program was positively associated with the availability of art 
activities, as reported by site coordinators (r=0.57, effect size=0.32). 

■ The extent to which participants reported that the program exposed them to new 
opportunities was positively associated with the availability of academically 
oriented activities that supported literacy skills development, cognitive 
development, and academic achievement, as reported by site coordinators 
(r=0.50, effect size=0.25). 



Levels of Activity Participation 

An analysis of activity and youth participation data reported by program staff in the NJ 
After 3 YouthServices.net database during 2007-08 revealed that, on average, participants 
engaged in: 



■ 54 hours of health and sports activities (e.g., nutrition, basketball, cooperative 
games) 

■ 41 hours of visual arts and perfonning arts activities (e.g., drawing, dance, drama) 

■ 38 hours of community service and character education activities (e.g., violence 
prevention, instruction in resisting peer pressure, food drives) 

■ 33 hours of literacy (e.g., reading groups, spelling bees, journal reflections) 

■ 32 hours of study skills and college exploration (e.g., test preparation, tutoring, 
trips to colleges) 

■ 25 hours of math and business activities (e.g., financial literacy, KidzMath) 

■ 19 hours of technology and science related activities (e.g., computer and video 
projects, earth science) 

■ 19 hours of cultural activities (e.g., politics, community exploration) 

■ 18 hours of “New Jersey” (e.g., New Jersey history, cultures, geography) 

These numbers of hours do not equal the total number of hours provided in NJ After 3 programs 
due to missing data in the Youthservices.net database and because programs can select up to five 
NJ After 3 categories to describe one activity in the database. (The database then divides the 
number of hours in which youth participate in an activity by the number of categories selected. 
For example, if a youth attends one activity for three hours and that activity has three categories 
selected, the database will assign one hour to each of these categories for that day.) 
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Activity Observations as a Window on Program Content 

An important feature of the evaluation was our structured observations of afterschool 
operations. The purpose of these observations was to record activity content and to rate activities 
on the extent to which they reflected standards of afterschool quality, as determined in particular 
by the research of Durlak and Weissberg (2007). Findings from these observations are 
summarized here. 

During site visits to the 10 in-depth sites, a team of eight PSA researchers conducted 78 
independent observations of program activities. A total of nine co-observations yielded an 
average inter-rater reliability (IRR) rating of 0.73, equivalent to the IRR in 2006-07. More than 
half of the 78 observed activities occurred in the classroom. Other commonly used activity 
spaces in which activities were observed included the cafeteria (17 percent) and gym (10 
percent). Each observed activity averaged approximately six girls and six boys and, overall, 
each activity averaged a total of 12 youth. On average, two staff members were present during 
each observed activity. Almost 60 percent of all observed activities were staffed with at least 
one college student or young adult; 37 percent of observed activities were staffed by at least one 
certified teacher. There were few high school students or content specialists in observed 
activities. Analyses found no statistically significant difference in the staff ratio by grade level 
(e.g., K-4 or 5-8). In most cases, youth were grouped in activities by age or grade (71 percent of 
observed activities), rather than by personal interest (26 percent). 

Most of the observed activities involved lower elementary students. Seventy-four percent 
of all observations involved kindergartners, first-, second-, third-, or fourth-graders. Thirty-five 
percent of all observations involved fifth-, sixth-, seventh-, or eighth- graders. All observed 
activities had appropriate adult supervision (100 percent, or 78 of 78 observations), and almost 
all were held in an appropriate work space (96 percent, or 75 of 78 observations) and involved 
sufficient materials (96 percent, or 75 of 78 observations). 

The three most frequently observed activities featured academic enrichment (33 percent 
of observations), visual and performing arts (31 percent), and homework help and tutoring (17 
percent). Twenty-three percent of the 78 observed activities focused on literacy. Among these 
activities, youth were often filling in a reading workbook, working on original writing, or 
reading aloud. Nine of the 78 observed activities (12 percent) focused on math. These activities 
often involved practicing basic math operations or working in workbooks as part of homework 
time. About 10 percent of all observed activities focused on technology, generally limited to 
using computers for internet searches. 

We grouped the scores for each indicator of youth development practice into scales. The 
scales were then aligned with key features of effective afterschool programs that were identified 
by Durlak and Weissberg in their meta-analysis of 73 afterschool programs (2007). These 
features, which are collectively referred to as the SAFE model, are outlined below: 



■ Sequenced (i.e., built skills and content to achieve goals) 

■ Involved the practice or a progression of skills 

■ Required analytic thinking 
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■ Challenged youth intellectually, creatively, developmentally, or physically 

■ Employed staff who used varied instructional strategies, provided 
assistance to youth without taking control, and verbally recognized youth 
efforts and accomplishments 

■ Active (i.e., offered opportunities to actively participate in learning) 

■ Provided opportunities for youth to collaborate, take leadership 
responsibilities and roles, make meaningful choices, assist each other, and 
contribute opinions and ideas 

■ Employed staff who encouraged youth to share their ideas, opinions, and 
concerns and who asked youth to expand upon their answers and ideas 

■ Focused (i.e., intended to develop positive relationships among youth and with 

staff) 

■ Included youth who showed positive affect to staff and were respectful, 
friendly, and relaxed with one another 

■ Employed staff who showed positive affect toward youth, engaged 
personally with youth, guided youth toward positive peer interactions, 
used positive behavior management techniques, and were equitable and 
inclusive in interactions with youth 

■ Explicit (i.e., targeted specific learning goals and/or developmental goals) 

■ Involved tasks and activities that were well organized 

■ Kept youth on task and attentive to peers and staff 

■ Employed staff who listened to and were attentive to youth 

Analysis of the data collected during structured observations of 78 afterschool 
activities — and that were subsequently aligned with the SAFE model — found that the mean 
scores on a scale from 1 (indicator is not evident) to 7 (indicator is highly evident and consistent) 
for each of the four SAFE domains were: 3.64 for “sequenced” practices that built skills and 
content knowledge to achieve goals; 1.83 for “active” practices that offered opportunities to 
actively participate in learning; 4.40 for “focused” practices that developed positive relationships 
among youth and with staff; and 5.21 for “explicit” practices that targeted specific learning goals 
and or developmental goals. The differences in scale scores between the evaluation’s second and 
third years were not statistically significant. 

Results of analysis of the scales by activity type are summarized in Exhibit 25. The 
analysis revealed that academic enrichment and sports activities had higher “sequenced” scores 
than did homework and open activities, indicating that these activities were more likely to be 
challenging and to involve a progression of skills. All activities tended to score low on the 
“active” scale, suggesting that programs provided few opportunities for hands-on practice and 
the application of skills. On the other hand, all activities scored high on the “focused” and 
“explicit” scales, indicating that activities were often structured to develop personal and social 
skills in youth and featured specific learning goals. 
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Exhibit 25 

Analysis of Program Observations, by SAFE Features and Activity Type 



Program 

Feature 


HW/Tutoring 

( n =13) 


Enrichment 

(n=26) 


Arts 

(n=24) 


Sports 

(n=12) 


Open 

(n = 3) 


Sequenced 


3.28 


4.04 


3.37 


3.87 


2.00 


Active 


1.56 


2.08 


1.69 


1.94 


1.54 


Focused 


4.14 


4.77 


4.17 


4.55 


3.58 


Explicit 


4.66 


5.77 


4.89 


5.47 


4.40 



Exhibit reads: The average “sequenced” scale score was 3.28 for homework/tutoring activities, 4.04 for 
enrichment activities, 3.37 for art activities, 3.87 for sports activities, and 2.00 for open activities. 



Analysis also showed that, in general, academic enrichment activities were significantly 
more likely to: involve a progression or practice of skills previously learned (averaging 4.04 points 
out of 7 on the “sequenced” scale, compared with 3.40 for all other activities); engage youth in 
making meaningful choices and applying skills (2.08 on the “active” scale, compared with 1.71 for 
all other activities); and to be well organized with clear learning goals (5.77 on the “explicit” scale 
compared with 4.94 for all other activities). The other significant difference among activities was 
between art activities (which averaged 4.17 points out of 7 on the “focused” scale) and all other 
activities (which averaged 4.5 1 on the same scale). This result indicates that, on average, art 
activities were less likely to provide opportunities for social and interrelationship development 
than were other activities. 



Communication with Parents 

In 2007-08, programs reached out to parents in various ways. Although many programs 
held events or invited parents to meetings, site coordinators reported that they made certain 
connections more frequently than others. At least once a month, all responding site coordinators 
(100 percent) said that they communicated with at least some parents over the phone. Ninety- 
three percent said that they met with one or more parents at least once a month. The nature and 
extent of communication with parents did not significantly differ from the first to the third years 
of the evaluation. However, in 2007-08, 91 percent of site coordinators said that they sent 
materials about the program home to parents, compared with 74 percent of site coordinators in 
2005-06 (effect size= 0.21), as shown in Exhibit 26. 

Parents appreciated the ways in which program staff responded to them and interacted 
with their children. In interviews, parents indicated that they were satisfied with the level of 
communication with NJ After 3 staff. “The staff here are very nice. They communicate with 
me. . said one parent. Parents also expressed satisfaction with staff interactions with their 
children. “They treat the kids kindly. They are good people here. They play with the kids, and 
let them play,” said one parent. “I ask the staff about what she does. . . [and ] I observe staff. 
They are very fair. They treat all the kids the same,” said another. “The staff, they’re really 
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supportive and very accommodating. They like the children. They help you as much as 
possible,” said a third. 



Exhibit 26 

Program Relationships with Parents and the Community, 
in Percents, 2005-06 and 2007-08 



At least once a month... 



Site coordinator has conversations with 
parents over the phone 



Site co o rdinato r meets with o ne o r mo re 
parents 



Site coordinator sends materials about the 
program home to parents* 



Site coordinator holds events or meetings 
to which parents are invited 



Site coordinator holds events or meetings 
to which community members are invited 




Percent of site coordinator survey respondents 



1 2005-06 (n=55) 



H 2007-08 (n=97) 



Exhibit reads: In 2007-08, 100 percent of site coordinators reported that they had conversations with parents over the 
phone at least once a month. In 2005-06, 90 percent of site coordinators reported that they had conversations with 
parents this frequently. This difference is not statistically significant. 

* Difference between years is significant at p<.05. 



41 



5. Accomplishments and Challenges 



In this section we highlight key accomplishments of the NJ After 3 initiative and examine 
challenges that have hindered program implementation and desired outcomes. 



Key Program Accomplishments 

The total number of youth enrolled in afterschool programs increased. Between 2004- 
OS and 2007-08, the number of youth served by NJ After 3 increased from approximately 4,000 
to 15,000 participants. 

The attendance rate of participating youth increased. Participants in NJ After 3 
attended more days in 2007-08 than did participants in previous years. Possible explanations for 
this trend are that, over time, programs have become more responsive to youth needs and 
interests. 

At sampled sites, NJ After 3 activities were found to emphasize relationship-building 
and social development and to target specific learning and developmental goals, all of which is 
consistent with recommendations of youth development research ( Little et al., 2008). Detailed 
observations of activities conducted at the 10 in-depth sites revealed that, on a 7-point scale, NJ 
After 3 activities were rated 4.40 for “focused” practices that developed positive relationships 
among youth and with staff and 5.21 for “explicit” practices that targeted specific learning goals 
and or developmental goals. 

Learning experiences in NJ After 3 programs may have positively influenced students’ 
study skills. The evaluation found that students who attended their NJ After 3 program most 
often were rated more highly by their teachers on the study-skills scale than were those who 
attended less often. (Other explanations of this difference are also plausible, however.) 

A majority of participants reported that they had a sense of attachment to the program 
and had benefited academically and socially from participating in it. Youth responded very 
positively to questions about their experiences in NJ After 3 (e.g., “staff care about me”) and a 
large majority were able to identify specific ways (e.g., homework completion, improvements in 
reading and writing) in which the afterschool program had helped them academically. Many 
also noted that the program had provided opportunities for new experiences. 

Parents reported that the program had allowed them to pursue employment and to 
work full-time rather than part-time schedules. All of the parents interviewed during each year 
of the evaluation spoke very positively about the program. In addition to identifying ways in 
which their children had benefited (e.g., increased confidence and improved social skills), many 
explained that the program allowed them to seek employment or increase their hours at work 
because they knew that their child was being well-supervised after school. 
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Challenges to Implementation and Program Quality 

Interview and survey data indicate that challenges related to student recruitment, 
curriculum and instruction, staffing and professional development, relationships between 
programs and host schools, and funding affected program implementation, expansion, and or 
improvement at several sites. The data suggest that these challenges touched some programs 
more than others. 

Some sites found it difficult to achieve attendance and retention goals. As illustrated in 
Exhibit 27, more than two-thirds of site coordinators reported that youth dropped out because 
they lost interest (68 percent) and that they could not recruit enough youth who wanted to 
participate (66 percent). In addition, almost three-quarters (74 percent) reported that youth do 
not attend their program regularly enough. As noted earlier in this report, New Jersey After 3 
staff are keenly focused on promoting very high attendance. While data presented previously 
indicate that youth in New Jersey After 3 programs attend program activities at comparatively 
high rates, site coordinators hold even higher attendance aspirations for participants. As the 
survey results in Exhibit 27 suggest, achieving and maintaining these high attendance rates pose 
challenges for a majority of site coordinators. 



Exhibit 27 

Site Coordinators’ Report of Major Challenges, in Percents, 2005-06 and 2007-08 



Issues Presenting a Major Challenge to Program Implementation 


2005-06 

(n=55) 


2007-08 

(n=97) 


We do not receive sufficient support or feedback from NJ After 3 


76 


83 


We do not have sufficient administrative support to fulfill NJ After 3 grant 
reporting requirements* 


62 


76 


Youth do not attend the NJ After 3 program regularly enough 


56 


74 


We have insufficient information about the needs of the participating youth 
in our program* 


49 


72 


Youth drop out because they lose interest 


70 


68 


We cannot recruit enough youth who want to participate 


67 


66 


We have inadequate instructional materials or programming ideas 


51 


61 


The space available for our program is inadequate, inappropriate, or unsafe 


47 


59 


The school(s) our participants attend would like our program to be more 
academically focused 


42 


54 


The staff at the school(s) participants attend do not respond to our requests 
to coordinate services or resources 


47 


53 


We do not have sufficient funds to provide high-quality programming 


53 


41 


Families are not sufficiently involved in their children’s participation at the 
program 


27 


28 



Exhibit reads: In 2007-08, 83 percent of site coordinators reported that insufficient support or feedback from NJ After 3 
presented a major challenge to program implementation. This difference is not statistically significant. 



* Difference between years is statistically significant at p<.05. 
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Some programs offered activities and classes that did not fully meet important goals 
and objectives. All or nearly all of the activities observed at the sampled sites met certain basic 
thresholds for quality (e.g., the maintenance of adequate staff/student ratios and the provision of 
adequate space and materials), but some programs offered activities and classes that failed to 
meet other important goals and objectives. Specifically, observed activities tended to score low 
on our “active” scale, suggesting that programs provided few opportunities for hands-on practice 
and the application of skills. Some site coordinators also reported dissatisfaction with program 
offerings. For example, as illustrated in Exhibit 27 above, in 2007-08, 61 percent reported that 
they had inadequate instructional materials or programming ideas, 59 percent reported that space 
available for the program was inadequate, inappropriate, or unsafe, and 41 percent said that they 
did not have sufficient funds to provide high-quality programming. 

Recruiting and retaining staff with desired skills, knowledge, and professionalism was 
difficult for some site coordinators . As Exhibit 28 illustrates, approximately half of all site 
coordinators said in 2007-08 that: staff do not have the skills to work with English-language 
learners (59 percent); there are limited professional development opportunities for staff (56 
percent); and that staff do not come to work on a reliable schedule (49 percent). 

Exhibit 28 

Major Staffing Challenges to Implementing High-Quality Programming, 
in Percents, 2006-07 and 2007-08 

Staffing challenges... 

Staff do not have the skills to wo rk with 
English-language learners 

There are limited professional development 
o ppo rtunities f o r staff 

Staff do not come to work on a reliable 
schedule 

Volunteers are not available on a reliable 
schedule 

We cannot afford to offer potential staff 
enough hours of paid employment 

We cannot find qualified staff to hire 

We cannot find volunteers with time and 

expertise needed to help in our program 

We cannot afford to offer the competitive 

salaries necessary to hire qualified staff* 

0 20 40 60 80 100 

Percent of site coordinator survey respondents 

■ 2005-06 (n=55) ■ 2007-08 (n=97) 



Exhibit reads: In 2007-08, 59 percent of site coordinators reported that staff not having the skills to work with 
English-language learners was a major staffing challenge, compared to 64 percent who reported this challenge 
in 2005-06. This difference is not statistically significant. 

* Difference between years was significant at p<.05. 
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Professional development for staff was an ongoing need. NJ After 3 and local partner 
organizations that host the afterschool programs provided various forms of professional development 
to staff to address perceived deficiencies and areas of need that site coordinators identified. 4 Site 
coordinators indicated that staff needed, and in many cases received, training in areas such as: 
managing classrooms and maintaining order, modeling appropriate behavior for youth, writing 
lesson plans, developing new activities for engaging students, providing differentiated instruction, 
recognizing child abuse, administering CPR, and communicating with parents. 

Several site coordinators insisted, however, that what staff needed most to be effective 
were less tangible skills, such as empathy, which are more difficult to cultivate. One site 
coordinator explained it this way: “I would like staff to have training on what exactly 
afterschool programming is all about. It is not babysitting, it’s about transforming lives. You 
can impact kids in a way that is not possible during the school day. [The children] have so many 
issues. . . I want my staff to get training in how to be more compassionate.” Similarly, another 
noted that working with children requires special training, and younger staff are not necessarily 
as prepared as they should be for this task. “There are a lot of things involved when you work 
with kids,” she explained. “You have to look at how they feel, how they are acting. . .notice what 
is going on with them. [You have to be able] to see behind why they are acting out. . Tittle things 
like that. It just comes with experience working with children.” 

A large majority (88 percent) of site coordinators reported that they or their staff 
participated in technical assistance or training through NJ After 3 in 2007-08. This percentage 
was, however, significantly lower that than the 98 percent who had reported that they or their 
staff received this training in 2005-06 and was the same level reported in 2006-07. Executive 
directors reported a trend that could have affected staff participation in training and professional 
development. In 2005-06, significantly more executive directors (77 percent, compared with 45 
percent in 2007-08) reported that, in addition to wages, the benefits they provided to full-time 
staff included “paid attendance at staff meetings and conferences” (effect size=0.30). 

Significantly fewer (83 percent in 2005-06 and 53 percent in 2007-08) also reported that they 
offered this benefit to part-time staff (effect size=0.30) over the course of the evaluation. 

Notwithstanding the opportunities available for training, many site coordinators reported 
that even more was required. They acknowledged that an unwillingness or inability on the part 
of some staff members to stay overtime or to travel to locations where training was available 
made it difficult for those who are most in need of assistance to actually receive the rigorous, in- 
depth training that they needed. 

Several programs reported poor communication and relationships with their host 
schools. As feedback from several site coordinators indicated, when relationships between the 
afterschool program and the host school were good, staff and youth benefited. 



4 NJ After 3 leadership has stated that NJ After 3 provides “over 80 hours of training and professional development 
to site coordinators each year, quadruple the state licensing requirement. Programs also receive two site visits from 
NJ After 3 headquarters staff every year with formal feedback following each visit.” 
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I bought a lot of Hooked on Phonics materials. I try to mimic what the teachers are 
doing [during the regular school clay]. I go sit in on classrooms every once in a while to 
see what they’re doing. [The teachers] let me know what’s needed. We work together. 

We ask [teachers] about changes in students’ behaviors [and] about how to handle 
certain situations. We ask the counselor if she has heard anything about students who 
seem to be having problems. With the principal we might ask, “We are doing [a 
particular activity] so may we use the auditorium?” [Our] relationship with the math 
and literacy coaches is ongoing. I may ccdl them and ask: “Can you give me activities 
[related to] letter recognition ? What test do [students] take? Are there worksheets on 
this topic?” 1 use the tests to guide what we do [in the after school program]. 

[Once] when [my students] were unclear about an assignment, their [regular] teacher 
came down and saw that most of the students from her class were around the table. She 
went over the steps with me, so I could help them better. They have new math [and there 
were ] some things I didn ’t understand. . . she helped me. ..[it was ] no problem. I like that. 
She gave me a positive response. 

On the other hand, poor relationships between program and school staff hindered the 
quality and effectiveness of afterschool programs by, for example, limiting staff and youth 
access to resources and information that could help improve programming. In 2007-08, 
approximately half (53 percent) of site coordinators reported that the staff at the host school(s) 
did not respond to their requests to coordinate services or resources. In addition, 72 percent of 
site coordinators said that insufficient infonnation about the needs of participating youth in their 
program was a major challenge. Significantly fewer (49 percent) identified this as a major 
challenge in 2005-06 (effect size=0.24). Similarly, survey results suggested a general decline 
between the evaluation’s first and third years in the percentage of site coordinators who reported 
that they discussed certain issues with school staff (e.g., discipline policies, curriculum concepts, 
state and local academic standards). 

Many programs apparently found it difficult to sustain themselves financially in the 
face of reduced support from NJ After 3. One site coordinator summed up the situation: “The 
decreasing budget year after year really kills us. We are not making up the difference.” The 
survey results suggest that, for at least some sites, this is generally accurate, if exaggerated: 
approximately 41 percent of site coordinators identified insufficient funds as a major challenge, 
and 40 percent identified it as a minor challenge. Another site coordinator presented the 
dilemma facing his program this way: 

Funding is a major priority but, because funding decreased every year, it ’s hard to 
provide the same level of sendee. We are trying to think of more resourceful ways of 
meeting needs. Initially, as partners, NJ After 3 was funding us [but the amount has 
graducdly decreased] . We have not gotten more funding from [our partner organization ] 
as NJ After 3 money decreased. If you spend cdl of your money hiring staff, [then there is 
little left for programming]. It’s hard to get quality staff without paying for [them] , and 
it’s hard to get quality programming because it costs money. We look for people to give 
us anything, pencils, crayons. ..We are not cdlowed to fundraise at our level. It must be 
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done at the corporate level. But [the partner organization] has so many programs, the 
money does not always trickle down to our program. 

A third site coordinator added that, even if time permitted, fundraising was difficult 
because of the relatively small size of the program. He explained that large businesses in the 
community tended to fund large programs, and the smaller businesses simply could not afford to 
make a meaningful contribution. 

As illustrated in Exhibit 29, while some site coordinators (25 percent to 73 percent) 
secured various types of assistance (e.g., volunteers or mentors, funding through grants or 
contracts, or donated materials or supplies) from outside organizations, the numbers in the 
exhibit revealed that many had not. 



Exhibit 29 

Support and Resources Received from External Organizations, in Percents, 

2005-06 and 2007-08 



Outside organizations... 

Provide special programs/activities/services 
for youth (on- or off-site) 

Donate materials or supplies 



P ro vide funding thro ugh grants o r co ntracts 

Refer students to the NJ After 3 afterschool 
program 

Provide regular volunteers or mentors 



Do nate facilities o r space 



Provide transportation services for students 



0 20 40 60 80 100 

Percent of site coordinator survey respondents 

■ 2005-06 (n=53) n 2007-08 (n=94) 




Exhibit reads: In 2007-08, 73 percent of site coordinators reported that at least one outside organization provided 
special programs, activities, or services for youth. In 2005-06, 75 percent of site coordinators responded similarly. 
The difference is not statistically significant. 



Executive directors also communicated that they had financial concerns as well. As 
illustrated in Exhibit 30, three of the top six challenges they identified (i.e., accessing funds, 
hiring qualified staff, and securing appropriate space and facilities) were directly or indirectly 
related to sustainability. Executive directors in 2007-08 reported that a lower percent of their 
organization’s NJ After 3 afterschool program budget came from “funds from state sources” than 
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were reported by executive directors in 2005-06. Executive directors in 2007-08 reported that 2 
percent came from state sources (excluding the state funds that are part of NJ After 3 funding), 
compared to 6 percent reporting that they received state funds in 2005-06. 



Exhibit 30 

Executive Directors’ Report of Major Challenges, in Percents, 2007-08 



Challenges for the partner organization... 



Managing administrative burden (e.g., paperwork, reports, 1 

pay roll processing) 

Accessing funds I 

Meeti n g en rol Iment and atten dance req uirements | 

Hiringqualifiedstaff I 

Balancing the demands of the NJ After 3 program with other 1 

o rg ani zational g oal s an d p rior iti es 1 

Securing appropriate spaceJacilities ~\ 

Meeti n g ti meli nes fo r p rogram start-up ~| 

Integ rating NJ After 3 funds with otherfunding streams I 

Meeting municipal or state licensing requirements | 

Usi n g th e Yo uthServices .net tracking system ~l 

■ i i i i 

0 20 40 60 80 100 

Percent of executive directors who reported this is a challenge "to a great extent" or "somewhat" 

(n=44) 



Exhibit shows: In 2007-08, 47 percent of executive directors reported that managing administrative burden was a 
challenge to implementing high-quality programming to a great extent or somewhat. 
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6. Recommendations 



NJ After 3 has achieved remarkable successes in a short space of time, as this report has 
described. The initiative has designed and implemented programming that serves large numbers of 
youth every afternoon that school is in session. Program partners and staff provide safe, appropriate 
settings for youth, from which youth derive apparent educational and social development benefits. 

To achieve these successes, NJ After 3 has worked hard to become and remain a learning 
organization. The following recommendation for continued organizational development are 
offered, based on our evaluation findings and on input offered directly by program staff. 

Reinforce the message to staff that fundraising is a priority and expand efforts to help 
programs become sustainable. Among site coordinators, 39 percent have written or helped to 
write one or more grant applications on behalf of the afterschool program, and 22 percent 
reported that they have solicited resources directly from individuals, organizations, or businesses. 
Over a quarter (27 percent) of site coordinators indicated that fundraising is not one of their 
responsibilities, and 36 percent of executive directors said that fundraising by site coordinators is 
not encouraged at their organization. NJ After 3 should work with individual sites to identify 
one or two individuals with whom they will work on issues related to fundraising and resource 
identification at each program. 

Individual programs can begin to sustain their own individual operations through at least 
four methods. The first is through earned income (e.g., fees charged to participants), the second 
is by attracting volunteers who provide free products or services (e.g., attracting in-kind 
donations), the third is philanthropy (e.g., monetary donations or grants from individuals or 
private institutions), and the fourth is government funds and programs. A plan that trains site 
coordinators and other program staff to secure assistance via all four strategies simultaneously 
and continuously may be most effective. 

Encourage sites to increase revenue through parent fees and other creative, 
entrepreneurial means. By introducing sliding scale fees rather than flat fees, programs may 
increase current revenue without placing undue financial burden on the most economically 
disadvantaged parents. Currently, only 36 percent of site coordinators indicated that they charge a 
fee for participation in the afterschool program, although comments from parents during the 
interviews suggest that some may be able and willing to pay a fee. Parents’ comments included the 
following: 

If it wasn 'tfor this program, we ’d have to pay someone to take care of [our children] 

because I [can 7 leave work early to pick them up from school]. 

[If this program was not available ] we’d have to pay a babysitter $100 a week. 

If my child was not in this program, he would be in another program [that cost more 

money], explained a site coordinator. 
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As fundraising has become more difficult, an increasing number of nonprofits have 
become more entrepreneurial, looking within existing program services and capacities for 
opportunities to generate needed income to support their programs. Individual sites and their 
respective host organizations could work with experts in the field of nonprofit management and 
social entrepreneurship to explore options for producing income-generating products or services 
that could provide resources to the afterschool program. 

Encourage sites to share resources. Sites that are in close proximity to each other might 
agree to share volunteer instructors and specialists, or benefit from economies of scale by sharing 
transportation costs (e.g., planning joint field trips). Assistance in securing new funding could 
include technical assistance in fonnal as well as informal fundraising methods (e.g., grant writing, 
strategies for securing meetings with local individuals and businesses, public speaking workshops, 
individualized brainstorming sessions regarding untapped resources in specific communities). 

Continue to help sites tap community resources. To address the related challenges of 
maximizing resources, staff qualifications, and program quality, NJ After 3 should help sites to, 
for example, identify free or inexpensive instructors and programs that are available through 
other nonprofit or government organizations. Such programs are often required to reach out to 
targeted groups (e.g., middle school youth) to provide a very specific service (e.g., financial 
literacy classes, golf or tennis instruction, CPR and first aid). A site coordinator told us that her 
program was developing a collaboration with a local university that requires some members of 
the freshman class to work for one year in a community-based organization. She noted that, 
because of this partnership, her afterschool program would receive staff assistance. 

Continue to build systems for nurturing relationships between programs and schools. 
Strong relationships between school-day and afterschool staff are an important contextual feature 
of afterschool programs that are located in schools. When relationships are good, instruction can 
be more targeted and responsive to participant needs, and additional space, materials, and other 
resources can be leveraged to enhance the scope and quality of afterschool classes and activities. 
Where relationships are poor, NJ After 3 can help site coordinators develop a plan of action (e.g., 
issue invitations to principals or district officials to attend a meeting or event; write a letter 
explaining why and how both parties can benefit from an improved relationship) to address the 
problem. As one site coordinator observed, “We need greater buy-in at the district level. We are 
thought of as another program, but we are really an extension of the school day. I don’t think 
that they understand that.” One strategy that may also be effective is to encourage site 
coordinators to hire school staff to work in the afterschool program. Although site coordinators 
provided mixed feedback during the site visits regarding the quality of school staff, it appears 
that in most cases the result of hiring current school personnel is greater communication between 
the organizations and increased access to resources and information for everyone. 

Take additional steps to increase interest in and access to professional development 
among staff who work directly with youth. Stipends, prizes, and other rewards for attendance, 
coupled with site-specific, targeted training might result in increased attendance at professional 
development events. Training sessions could also be specifically informed by, and be designed to 
address, specific and perhaps unique needs that are expressed by small groups of sites. Also, needs 
that have been identified by this evaluation (e.g., strategies to engage youth in more active, hands- 
on learning) could be among the topics offered. 
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Appendix A 

Details of Data Used in Analyses 
of Participant Experiences 




Participant Survey Scales 



Exposure to New Opportunities 

The Exposure to New Opportunities scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

In this afterschool program. . . 

■ I get a chance to do a lot of new things 

■ I get to do things that I don’t usually get to do anywhere else 

■ I get to work on activities that really make me thi nk 

■ There is a lot for me to choose to do 

■ The activities really get me interested 

■ I get to do community service projects 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.78 


2.92 


0.72 


1 


2.50 


3.50 


4 
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Community Service Projects 

The Community Service Projects scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

Participating in community service projects with this afterschool program, I feel like. . . 

■ My actions make a difference in the community 

■ My actions help others 

■ I have learned more about my community 

■ I have learned more about how I can help others 

■ I have learned more about other organizations in my community 

■ It is important to volunteer and help others 

■ I will continue to volunteer to help others in my community 

■ I can call myself a volunteer 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.90 


2.92 


0.85 


1 


2.50 


3.62 


4 
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Sense of Belonging 

The Sense of Belonging scale was computed to range from one to four, with four indicating that 
on average participants strongly agreed with the following statements: 

In this afterschool program, I feel like. . . 

■ I belong 

■ My ideas count 

■ lam successful 

■ This is a good place to hang out 

■ I matter 

■ I am safe 

■ My actions make a difference 

■ I have the power to help others 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.86 


3.24 


0.70 


1 


2.88 


3.88 


4 
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1.00 1.01 - 1.50 1.51 - 2.00 2.01 - 2.50 2.51 - 3.00 3.01 - 3.50 3.51- 4.00 

Scale score 

Disagree a lot Agree a lot 

■ Year 1 (n=640) aYear3 (n=935) 
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Interactions with Staff 

The Interactions with Staff scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In this afterschool program. . . 

■ Staff treat me with respect 

■ I feel that I can talk to staff about things that are bothering me 

■ Staff really care about me 

■ Staff often keep their promises 

■ Staff care what I thi nk 

■ Staff try to be fair 

■ Staff think I can do things well 

■ Staff help me to try new things 

■ Staff think I can leam new things 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.91 


3.19 


0.75 


1 


2.78 


3.78 


4 




Disagree a lot 



Scale score 



■ Year 1 (n=638) ■ Year 3 (n=942) 



Agree a lot 
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Interactions with Peers 



The Interactions with Peers scale was computed to range from one to four, with four indicating 
that on average participants strongly agreed with the following statements: 

In this afterschool program, I. . . 

■ Get to know other kids really well 

■ Can really trust the other kids 

■ Have a lot of friends 

■ Like the other kids 

■ Have a good time playing with other kids 

■ Get along with other kids 

■ Get to work with other kids as part of a team 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.85 


3.23 


0.68 


1 


2.86 


3.80 


4 
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Appendix B 

Details of Data Used in Analyses 
of Participant Academic Engagement and Skills 




Exhibit B1 

Teacher Reports of Participants’ Interpersonal Skills, Grades 5-8, 2007-08 



Gets alo n g with peopl e who are d i fferent 
Followsclassroom rules 
Accepts suggestions from teachers 
Interacts appropriately with adults 
Co rrects i n appropriate behavior wh en asked 
Works effectively in a small group activity 
Listens to what others have to say 
Interacts appropriately with other students 
Works effectively in a large group activity 
Expresses dissatisfaction appropriately 

0 20 40 60 80 100 

Percent of participants as reported by teachers 

(n=209) 



Exhibit reads: According to teachers, 75 percent of participants in grades 5-8 “almost always” or “often” got along with 
people who were different. 
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Exhibit B2 

Teacher Reports of Participants’ Academic Engagement, Grades 5-8, 2007-08 



Speaks in class when called upon 
Asks q uestions abo ut test o r p rojects 
Participates inclassdiscussions 
Volunteers answers to questions 
Initiates conversations appropriately 
Asks q uestions when confused 
Volunteers to read aloud 
Assumes Leadership in group situations 

0 20 40 60 80 100 

Percent of participants as reported by teachers 

(n=208) 



Exhibit reads: According to teachers, 77 percent of participants in grades 5-8 “almost always” or “often” spoke in 
class when called upon. 
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Exhibit B3 

Teacher Reports of Participants’ Academic Motivation, Grades 5-8, 2007-08 




Exhibit reads: According to teachers, 62 percent of participants in 5-8 “almost always” or “often” were motivated to 
learn. 
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Exhibit B4 

Teacher Reports of Participants’ Study Skills, Grades 5-8, 2007-08 




Patent of |»rticj|>mtt as imported bj, teachers 

(n=207) 



Exhibit reads: According to teachers, 74 percent of participants in grades 5-8 “almost always” or “often” took care of 
materials. 



B-4 



Exhibit B5 

Teacher Reports of Participants’ Reading and Language Arts Skills, 

Grades 5-8, 2007-08 



Oral communication 
Identifying a main idea 
Spelling 

Drawing conclusions from written material 
Punctuation 
Read i n g comprehension 
Vocabulary 
Written communication 
Reading fluency 
Word attack 
Grammar 




Percent of participants as reported by teachers 

(n=209) 



Exhibit reads: According to teachers, 83 percent of participants in grades 5-8 met at least grade-level expectations for 
oral communication skills. 
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Exhibit B6 

Teacher Reports of Participants’ Technology Skills, Grades 5-8, 2007-08 



Playing games 



Useofthe Internetfor 
research 



Useofword processing 
program 



Sen d i ng an d receivi ng emai I 



Use of spreadsheet program 




0 20 40 60 80 100 



Percent of participants as reported by teachers 

(n =1 76) 



Exhibit reads: According to teachers, 81 percent of participants in grades 5-8 exhibited “excellent” or “good” 
technology skills in playing games. 
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Exhibit B7 

Teacher Reports of Frequency of Communication with Parents, 

Grades 5-8, 2007-08 



In addition to scheduled report cards, I send written progress 
reports to this student's parents/guardians 

This student's parent/guardian attends school-scheduled 
p aren t-teacher co nferences 

In addition to school-scheduled parent-teacherconferences, 
I tal k i n p erson wi th th i s student's p arents abo ut th is . . 

Icon tact th i s student's parents/guardians byphoneoremail 

I involve this student's parents/guardians in planning new 
programming to match this student's needs 

Th is student's parents/guardians contact me by phone or 
email 



0 20 40 60 80 100 

Percent of teachers who reported "often" or "sometimes" 

(n=208) 



Exhibit reads: Eighty-seven percent of responding teachers reported that, in addition to scheduled report cards, they 
sent written progress reports to the student’s parents/guardians. 
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Teacher Survey Scales 

Interpersonal Skills 

Interpersonal skills indicate how well students follow rules, accept limits, and interact with 
adults and their peers. The Interpersonal Skills scale was computed to range from one to five, 
with five indicating that on average participants almost always exhibited the following skills: 

■ Follows classroom rules 

■ Corrects inappropriate behavior when asked 

■ Expresses dissatisfaction appropriately 

■ Accepts suggestions from teachers 

■ Works effectively in a large group activity 

■ Interacts appropriately with adults 

■ Listens to what others have to say 

■ Gets along with people who are different 

■ Works effectively in a small group activity 

■ Interacts appropriately with other students 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.98 


3.96 


0.77 


1 


3.50 


4.70 


5 
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Academic Engagement 

Academic engagement indicates students’ willingness to volunteer to answer questions, ability to 
assume leadership, and readiness to participate in class discussions. The Academic Engagement 
scale was computed to range from one to five, with five indicating that on average participants 
almost always exhibited the following skills: 

■ Speaks in class when called upon 

■ Asks questions about tests or projects 

■ Participates in class discussions 

■ Volunteers answers to questions 

■ Assumes leadership in group situations 

■ Volunteers to read aloud 

■ Initiates conversations appropriately 

■ Asks questions when confused 



Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.96 


3.75 


0.86 


1 


3.00 


4.38 


5 
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Never 



2.51 
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Scale score 



3.01 - 
3.50 



3.51 - 
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4.51 - 
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Academic Motivation 



Academic motivation indicates students’ eagerness to leam, willingness to take on challenges, 
ability to stay on target, and responsibility for their own learning. The Academic Motivation 
scale was computed to range from one to five, with five indicating that on average participants 
almost always exhibited the following skills: 

■ Is motivated to learn 

■ Prefers challenging tasks 

■ Produces high-quality work 

■ Critically evaluates own work 

■ Attempts to improve on previous performance 

■ Makes the most of learning experiences 

■ Persists when task is difficult 

■ Looks for ways to academically challenge self 

■ Assumes responsibility for own learning 

■ Is goal-oriented 

■ Stays on task 



Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.98 


3.53 


.94 


1 


3.00 


4.00 


5 
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Study Skills 



Study skills illustrate how well students prepare for tests, how often they complete homework, 
and how often they correct their own work. The Study Skills scale was computed to range from 
one to five, with five indicating that on average participants almost always exhibited the 
following skills: 



■ Completes homework 

■ Corrects own work 

■ Finishes class work on time 

■ Prepares for tests 

■ Prepares for class 

■ Turns in homework on time 

■ Takes care of materials 

■ Pays attention in class 

■ Completes assignments according to directions 

■ Takes notes in class 

■ Reviews materials 



Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.98 


3.84 


0.82 


1 


3.00 


4.00 


5 
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Reading and Language Arts 

Reading and language arts skills denote students’ ability to identify a main idea, use grammar 
and punctuation correctly, and draw conclusions from written material. The Reading and 
Language Arts Skills scale was computed to range from one to five, with five indicating that on 
average participant reading and language arts skills were far above grade-level expectations for 
the following skills: 

■ Reading comprehension 

■ Word- attack 

■ Vocabulary 

■ Identifying a main idea 

■ Reading fluency 

■ Spelling 

■ Punctuation 

■ Grammar 

■ Written communication 

■ Oral communication 

■ Drawing conclusions from written material 



Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.99 


2.97 


0.80 


1 


2.45 


3.00 


5 
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Technology Skills 

Technology skills refer to students’ ability to use word processing programs, use the Internet for 
research, and send and receive e-mails. The Technology Skills scale was computed to range 
from one to four, with four indicating that on average participants technology skills were 
excellent for the following indicators: 

■ Use of word processing program 

■ Use of spreadsheet program 

■ Use of Internet for research 

■ Sending and receiving e-mail 

■ Playing games 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.93 


3.00 


0.66 


1 


2.80 


3.40 


4 
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Communication with Parents 



Research suggests that higher parental involvement will yield more positive results in children’s 
academic skills and success. The Communication with Parents scale was computed to range 
from one to four, with four indicating that on average teachers often communicated with parents 
in the following ways: 

■ This student’s parents/guardian attends school-scheduled parent-teachers conferences. 

■ In addition to school-scheduled parent-teacher conferences, I talk in person with this 
student’s parents/guardians about this student’s difficulties and/or progress. 

■ I contact this student’s parents/guardians by phone or e-mail. 

■ This student’s parents/guardians contact me by phone or e-mail. 

■ In addition to scheduled report cards, I send written progress reports to this student’s 
parents/ guardians . 

■ I involve this student’s parents/guardians in planning new programming to match this 
student’s needs. 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.85 


2.86 


0.69 


1 


2.50 


3.17 


4 
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Participant Survey Scale 

Academic Benefits of the Program 

The Academic Benefits of the Program scale was computed to range from one to four, with four 
indicating that on average participants strongly agreed with the following statements: 

The afterschool program has helped me. . . 

■ Get better grades in school 

■ Feel better about my schoolwork 

■ Read and understand better 

■ Solve math problems better 

■ Finish my homework more often 

■ Write better 

■ Use computers to do schoolwork better 
Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.86 


2.92 


0.79 


1 


2.42 


3.57 


4 
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Appendix C 

Details of Data Used in Program Observation Analyses 




Evaluators conducted seven to nine activity observations in each of the 12 in-depth study 
sites. Evaluators used PSA’s OST Observation Instrument to conduct these structured 15 minute 
observations. In total, observation data represent 78 independent observations and nine activity 
co-observations, with an average inter-rater reliability of 0.727. Each observation indicator was 
rated from one to seven, with seven meaning that the indicator was highly evident and consistent 
throughout the observation. The four scales described below are based on the SAFE (Sequenced, 
Active, Focused, Explicit) model of program quality features identified by Durlak and Weissburg 
(2007) in their meta-analysis of afterschool programs. 
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Sequenced : The activity builds on skills and content to achieve goals. 

The Sequenced scale combines ratings from the following indicators: 

■ Activity involves the practice or a progression of skills 

■ Staff challenges youth to move beyond their current level of competency 

■ Activity requires analytical thinking 

■ Staff employs varied teaching strategies 

■ Activity challenges students intellectually, creatively, developmentally, and/or 
physically 

■ Staff assists youth without taking control 

■ Staff verbally recognizes youth efforts and accomplishments 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.77 


3.65 


1.12 


1.00 


2.79 


4.50 


5.71 
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Active : The activity offers youth opportunities to actively participate in learning. 
The Active Programming scale combines ratings from the following indicators: 

■ Staff plan for and ask youth to work together 

■ Youth are collaborative 

■ Youth take leadership responsibilities and roles. 

■ Youth have opportunities to make meaningful choices 

■ Youth assist one another 

■ Youth contribute opinions, ideas and concerns to discussions 

■ Staff encourages youth to share their ideas, opinions and concerns 

■ Staff asks youth to expand upon their answers and ideas 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.67 


1.83 


0.83 


1.00 


1.09 


2.38 


4.13 
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Focused : The activity focuses on developing positive relationships among youth and with staff. 
The Focused Activity scale combines ratings from the following indicators: 

■ Youth show positive affect to staff 

■ Youth are friendly and relaxed with one another 

■ Youth respect one another 

■ Staff shows positive affect toward youth 

■ Staff guides for positive peer interactions 

■ Staff uses positive behavior management techniques 

■ Staff is equitable and inclusive 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.76 


4.40 


0.68 


2.75 


4.00 


4.88 


6.00 
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Explicit ; The activity explicitly targets specific learning and social development goals. 
The Explicit scale combines ratings from the following indicators; 

■ Activity is well organized 

■ Youth are on task 

■ Staff communicates goals, purposes, and expectations 

■ Y outh listen actively and attentively to peers and staff 

■ Staff attentively listens to and observes youth 

Descriptive Statistics: 



Alpha 


Mean 


Standard 

Deviation 


Minimum 


25 th 

Percentile 


75 th 

Percentile 


Maximum 


0.85 


5.22 


0.89 


2.60 


4.60 


5.85 


7.00 
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